# [Alexa 2004] Point-Based Computer Graphics (Course Notes).pdf

Point-Based Computer Graphics SIGGRAPH 2004 Course Notes Marc Alexa, Darmstadt University of Technology Markus Gross, ETH Zurich Mark Pauly, Stanford University Hanspeter Pfister, Mitsubishi Electric Research Laboratories Marc Stamminger, University of Erlangen-Nuremberg Matthias Zwicker, Massachusetts Institute of Technology Abstract This course introduces points as a powerful and versatile graphics primitive. Speakers present their latest concepts for the acquisition, representation, modeling, processing, and rendering of point sampled geometry along with applications and research directions. We describe algorithms and discuss current problems and limitations, covering important aspects of point based graphics. C O N T E N T S Section 1: Introduction Section 2: Course Schedule Section 3: Talk Slides Section 4: Supplemental Material S E C T I O N 1 INTRODUCTION 1 Point primitives have experienced a major "renaissance" in recent years, and considerable research has been devoted to efficient representation, modeling, processing, and rendering of point-sampled geometry. There are two main reasons for this new interest in points: First, we have witnessed a dramatic increase in the polygonal complexity of computer graphics models. The overhead of managing, processing, and manipulating very large polygonal-mesh connectivity information has led many leading researchers to question the future utility of polygons as the fundamental graphics primitive. Second, modern 3D digital photography and 3D scanning systems acquire both the geometry and the appearance of complex, real-world objects. These techniques generate huge volumes of point samples, which constitute discrete building blocks of 3D object geometry and appearance, much as pixels are the digital elements for images. This course presents the latest research results in point-based computer graphics. After an overview of the key research issues, affordable 3D scanning devices are discussed, and novel concepts for mathematical representation of point-sampled shapes are presented. The course describes methods for high-performance and high-quality rendering of point models, including advanced shading, antialiasing, and transparency. It also presents efficient data structures for hierarchical rendering on modern graphics processors and summarizes methods for geometric processing, filtering, and resampling of point models. Other topics include: a framework for shape modeling of point-sampled geometry, including Boolean operations and free-form deformations, and Pointshop3D, an open-source framework that facilitates design of new algorithms for point-based graphics. 1.1 PREREQUISITES The course assumes familiarity with the standard computer graphics techniques for surface representation, modeling, and rendering. No previous knowledge about point-based methods is required. 1.2 INTENDED AUDIENCE The course is intended for computer graphics researchers who would like to learn more about point based techniques. They will obtain a state-of-the-art overview of the use of points to solve fundamental computer graphics problems such as surface data acquisition, representation, processing, modeling, and rendering. With this course, we hope to stimulate research in this exciting field. To this aim we will also introduce Pointshop3D, an open source software package that is publicly available as a test-bed for point-based computer graphics research. 1.3 COURSE SYLLABUS Course Introduction. Gross (15 minutes) Acquisition of Point Sampled Geometry and Appearance. Pfister (60 minutes) System overview Silhouette extraction View-dependent hull calculation Point-sampled data structure View-dependent shading Results Point Based Surface Representations. Alexa (60 minutes) Introduction & basics Delaunay subcomplex methods Approximating or interpolating implicits Projection-based methods Algorithms for High Quality Point Rendering. Zwicker (60 minutes) Signal processing basics and antialiasing Resampling filters for point rendering Surface splatting: a splatting algorithm with antialiasing Hardware acceleration Effiecient Data Structures. Stamminger (45 minutes) The QSplat bounding spheres hierarchy Optimizing the memory layout for hardware accelerated rendering Sequential point trees Implementation and results Processing, Sampling and Filtering of Point Models. Gross (30 minutes) Spectral processing of point sampled geometry Decomposition and patching of point clouds Local and global Fourier transforms Advanced geometric filtering Error and resampling Efficient Simplification of Point Sampled-Geometry. Pauly (30 minutes) Local surface analysis Simplification methods Hierarchical clustering, iterative simplification, particle simulation Comparison and analysis Pointshop3D: An Interactive System for Point-Based Surface Editing. Gross (30 minutes) System overview Point cloud parameterization Surface reconstruction Editing operators (painting, texturing, filtering, sculpting) Interactive demo Shape Modeling of Point-Sampled Geometry. Pauly (30 minutes) Boolean operations Free-form deformation Pointshop3D Demonstration. Pauly (30 minutes) Open system architecture Demonstration of interactive surface editing and modeling Panel on the Future of Point Based Computer Graphics. all (30 minutes) 1.4 COURSE WEBSITE We have compiled a web site for the course including slides and presentation material, which can be found on http://graphics.ethz.ch/publications/tutorials/points/. The Pointshop3D software package, including source code and documentation, is publicly available from the Pointshop3D web site http://graphics.ethz.ch/pointshop3d/. Using the software infrastructure provided by Pointshop3D course attendants will be able to quickly prototype their research ideas based on point primitives. 1.5 SPEAKERS CONTACT INFORMATION Dr. Marc Alexa Professor Department of Computer Science Darmstadt University of Technology Fraunhoferstr. 5 64283 Darmstadt Germany [email protected] http://www.igd.fhg.de/~alexa Dr. Markus Gross Professor Department of Computer Science Swiss Federal Institute of Technology (ETH) CH 8092 Zürich Switzerland [email protected] http://graphics.ethz.ch Dr. Mark Pauly Postdoctoral Associate Computer Science Department Gates Building 375 Stanford University Stanford, CA 94305 USA [email protected] http://graphics.stanford.edu/~mapauly/ Dr. Hanspeter Pfister Associate Director MERL - Mitsubishi Electric Research Laboratories 201 Broadway Cambridge, MA 02139 USA [email protected] http://www.merl.com/people/pfister/ Dr. Marc Stamminger Professor Graphische Datenverarbeitung Erlangen University of Erlangen-Nurnberg Am Weichselgarten 9 91058 Erlangen Germany [email protected] http://www9.informatik.uni-erlangen.de/Persons/Stamminger Dr. Matthias Zwicker Postdoctoral Associate The Stata Center, 32 Vassar Street Massachusetts Institute of Technology Cambridge, MA 02139 USA [email protected] http://graphics.csail.mit.edu/~matthias 1.6 SPEAKERS BIOGRAPHIES Marc Alexa. is an assistant professor at Darmstadt University of Technology and head of the Discrete Geometric Modeling group. He received his PhD and MS degrees in Computer Science with honors from Darmstadt University of Technology. His research interests include shape representations, modeling, transformation and animation as well as conversational user interfaces and information visualization. Markus Gross . Markus Gross is a professor of computer science and the director of the computer graphics laboratory of the Swiss Federal Institute of Technology (ETH) in Zürich since 1994. He received a degree in electrical and computer engineering and a Ph.D. on computer graphics and image analysis, both from the University of Saarbrucken, Germany. From 1990 to 1994 Dr. Gross worked for the Computer Graphics Center in Darmstadt, where he established and directed the Visual Computing Group. His research interests include point based graphics, physics-based modeling, multiresolution analysis and virtual reality. He has widely published and lectured on computer graphics and scientific visualization and he authored the book "Visual Computing", Springer, 1994. Dr. Gross has taught courses at major graphics conferences including SIGGRAPH, IEEE Visualization, and Eurographics. He is associate editor of the IEEE Computer Graphics and Applications and has served as a member of international program committees of many graphics conferences. Dr. Gross was a papers co-chair of the IEEE Visualization '99, the Eurographics 2000, and the IEEE Visualization 2002 conferences. Dr. Gross is a member of the ETH research and planning commissions. Mark Pauly. Mark Pauly obtained his PhD from the Computer Graphics Lab at ETH Zurich, Switzerland. Before, he had received his MS degree in computer science (with honors) from the University of Kaiserslautern, Germany. He has been working on point-based surface representations for 3D digital geometry processing, focusing on spectral methods for surface filtering and resampling. Further research activities are directed towards multiresolution modeling, geometry compression and texture synthesis of point-sampled objects. Currently, Dr. Pauly is a postdoctoral scholar with Stanford University. Hanspeter Pfister . Hanspeter Pfister is Associate Director and Senior Research Scientist at MERL - Mitsubishi Electric Research Laboratories - in Cambridge, MA. He is the chief architect of VolumePro, Mitsubishi Electric's real-time volume rendering hardware for PCs. His research interests include computer graphics, scientific visualization, and computer architecture. His work spans a range of topics, including point-based graphics, 3D photography, volume graphics, and computer graphics hardware. Hanspeter Pfister received his Ph.D. in Computer Science in 1996 from the State University of New York at Stony Brook. He received his M.S. in Electrical Engineering from the Swiss Federal Institute of Technology (ETH) Zurich, Switzerland, in 1991. Dr. Pfister has taught courses at major graphics conferences including SIGGRAPH, IEEE Visualization, and Eurographics. He is Associate Editor of the IEEE Transactions on Visualization and Computer Graphics (TVCG), member of the Executive Committee of the IEEE Technical Committee on Graphics and Visualization (TCVG), and has served as a member of international program committees of major graphics conferences. Dr. Pfister was the general chair of the IEEE Visualization 2002 conference in Boston. He is member of the ACM, ACM SIGGRAPH, IEEE, the IEEE Computer Society, and the Eurographics Association. Marc Stamminger. Marc Stamminger received his PhD in computer graphics in 1999 from the University of Erlangen, Germany, for his work about finite element methods for global illumination computations. After that he worked as a PostDoc at the Max-PlanckInstitut for Computer Science (MPII) in Saarbrücken, Germany and in Sophia -Antipolis in France on interactive rendering and modeling of natural environments. Since 2002 he is a professor for computer graphics and visualization at the University of Erlangen. His current research interests are pointbased methods for complex, dynamic scenes, and interactive global illumination methods. Matthias Zwicker. Matthias Zwicker obtained his PhD from the Computer Graphics Lab at ETH Zurich, Switzerland. He has developed rendering algorithms and data structures for point-based surface representations, which he presented in the papers sessions of SIGGRAPH 2000 and 2001. He has also extended this work towards high quality volume rendering. Other research interests concern compression of point-based data structures, acquisition of real world objects, and texturing and painting of point sampled surfaces. Currently, Dr. Zwicker is a postdoctoral associate with the Computer Graphics Group at MIT. 2 S E C T I O N COURSE SCHEDULE 2 TABLE 2.1 Morning Schedule Time Topic Lectureer 8.30-8.45 Welcome and Introduction Gross 8.45-9.45 Acquisition Pfister 9.45-10.15 Representations – I Alexa 10.15-10.30 Coffee Break 10.30-11.00 Representations – II Alexa 11.00-12.15 High Quality Rendering Zwicker 12.15-1.45 Lunch Break TABLE 2.2 Afternoon Schedule Time Topic Speaker 1.45-2.30 Data Structures Stamminger 2.30-3.00 Processing and Filtering Gross 3.00-3.30 Simplification Pauly 3.30-3.45 Coffee Break 3.45-4.15 Pointshop3D Gross 4.15-4.45 Shape Modeling (Pointshop 2) Pauly 4.45-5.15 Pointshop3D Demo Pauly 5.15-5.30 Panel all 3 S E C T I O N 3 TALK SLIDES Course Introduction. Gross Acquisition of Point Sampled Geometry and Appearance. H. Pfister Point Based Surface Representations. M. Alexa Algorithms for High Quality Point Rendering. M. Zwicker Effiecient Data Structures. M. Stamminger Processing, Sampling and Filtering of Point Models. M. Gross Efficient Simplification of Point Sampled-Geometry. M. Pauly Pointshop3D: An Interactive System for Point-Based Surface Editing. M. Gross Shape Modeling of Point-Sampled Geometry. M. Pauly So…Points • Point primitives have experienced a major „renaissance“ in Graphics • Two reasons for that: Point-Based Computer Graphics Marc Alexa, Markus Gross, Mark Pauly, Hanspeter Pfister, Marc Stamminger, Matthias Zwicker Polynomials.... – Dramatic increase in polygonal complexity – Upcoming 3D digital photography • Researchers start to question the utility of polygons as „the one and only“ fundamental graphics primitive • Points complement triangles ! Polynomials -> Triangles • Piecewise linear approximations • Irregular sampling of the surface • No parameterization needed (for now) 9Rigorous mathematical concept 9Robust evaluation of geometric entities 9Shape control for smooth shapes 2 Require proper parameterization 2 Discontinuity modeling 2 Topological flexibility Triangle meshes • Multiresolution modeling • Compression • Geometric signal processing Reduce p, refine h! Triangles... Triangles -> Points 9 Simple and efficient representation 9 Hardware pipelines support ∆ 9 Advanced geometric processing 9 The widely accepted queen of graphics primitives 2 2 2 2 Sophisticated modeling is difficult (Local) parameterizations still needed Complex LOD management Compression and streaming is highly non-trivial Remove connectivity ! • • • • Piecewise linear functions to Delta distributions Discrete samples of geometry No connectivity or topology – most simple Store all attributes per surface sample Point clouds • Points are natural representations for 3D acquisition systems • Meshes constitute an „enhancement“ of point samples 1 (Incomplete) History of Points – – – – – – – – – – – – – – – – – – – – – – A Motivation… Particle systems [Reeves 1983] Points as a display primitive [Whitted, Levoy 1985] Oriented particles [Szeliski, Tonnesen 1992] Particles and implicit surfaces [Witkin, Heckbert 1994] Rendering Architectures [Grossmann, Dally 1998] Digital Michelangelo [Levoy et al. 2000] Image based visual hulls [Matusik 2000] Surfels [Pfister et al. 2000] QSplat [Rusinkiewicz, Levoy 2000] Point Clouds [Linsen, Prautzsch 2001] Point set surfaces [Alexa et al. 2001] Radial basis functions [Carr et al. 2001] Surface splatting [Zwicker et al. 2001] Randomized z-buffer [Wand et al. 2001] Sampling [Stamminger, Drettakis 2001] Opacity hulls [Matusik et al. 2002] Pointshop3D [Zwicker, Pauly, Knoll, Gross 2002] Raytracing [Alexa et al. 2003] Sequential Point Trees [Dachsbacher, Stamminger 2003] Boolean Operations [Adams et al. 2003] Modeling [Pauly et al. 2003] blue-c [Gross et al. 2003] • 3D content creation pipeline Points generalize Pixels ! Taxonomy Schedule - Morning Rendering (Zwicker) Acquisition (Pfister) Point-Based Graphics Representation (Alexa) Time Topic Lecturer 8.30-8.45 Welcome and Introduction Gross 8.45-9.45 Acquisition Pfister 9.45-10.15 Representations – I Alexa 10.15-10.30 Coffee Break Processing & Editing (Gross, Pauly) Schedule - Afternoon 10.30-11.00 Representations – II Alexa 11.00-12.15 High Quality Rendering Zwicker 12.15-1.45 Lunch Break The Purpose of our Course is … Time Topic Lecturer 1.45-2.30 Data Structures Stamminger 2.30-3.00 Processing and Filtering Gross 3.00-3.30 Simplification Pauly 3.30-3.45 Coffee Break 3.45-4.15 Pointshop3D 4.15-4.45 Shape Modeling (Pointshop 2) Pauly Gross 4.45-5.15 Pointshop3D Demo Pauly 5.15-5.30 Panel on the future of PBG All 1. …to introduce points as a versatile and powerful graphics primitive 2. …to present state of the art concepts for acquisition, representation, processing and rendering of point sampled geometry 3. …to stimulate YOU to help us to further develop Point Based Graphics 2 Acquisition of Point-Sampled Geometry and Appearance Hanspeter Pfister, MERL [[email protected]] In this part of the course we will discuss acquisition and rendering of point-sampled models from real-world objects. 1 In Collaboration With • • • • • • Wojciech Matusik, MIT / MERL Addy Ngan, MIT Matt Loper, MERL Paul Beardsley, MERL Remo Ziegler, MERL Leonard McMillan, UNC This work would not have been possible without the people mentioned on this slide. In particular, the hard work and great talent of Wojciech Matusik was instrumental for this research. 2 How can we capture reality? The goal of our work is – quite simply – to capture and display complex threedimensional objects, such as an angle with feathers, a bonsai tree, or a teddy bear. A secondary goal is to solve one of the longstanding open problems in computer graphics: capturing and rendering a glass of beer. 3 Goals • Fully-automated 3D model creation • Faithful representation of appearance • Placement into new virtual environments More specific goals are to build a system with fully-automated 3D model acquisition. We focus on faithful acquisition of appearance, not geometry, although good geometry will help our rendering results. Our objects can be placed in arbitrary environments with arbitrary lighting and can be rendered from any viewpoint. The pictures show some of the acquired models (hat, ship, glass) placed into new environments. 4 Previous Work - I • Contact digitizers – intensive manual labor • Passive methods – require texture, Lambertian BRDF • Active light imaging systems – restrict types of materials • Fuzzy, transparent, and refractive objects are difficult 4 million pts. [Levoy et al. 2000] There has been a lot work on acquiring high quality 3D shape of real-world objects. This includes contact digitizers, passive methods, and active light imaging systems. Contact digitizers require a lot of manual labor. Nearly all passive techniques require that the object has texture and that the BRDF of the object is Lambertian. Active Methods are very popular. For example, on the right you see a laser rangescan of David, one of the great point-based models that resulted from the Digital Michelangelo project at Stanford [Levoy et al. 2000]. However, active light systems place severe restrictions on types of materials that can be scanned. For example, specular and fuzzy materials are difficult to scan. Furthermore, the obtained models require complicated alignment and hole filling procedures. 5 Previous Work - II • BRDF estimation, inverse rendering • Image based modeling • Point-based rendering There has been also a considerable amount of work on estimation of appearance of objects from a set of images. These methods usually try to fit parametric BRDF model to the observed measurements. However parametric BRDF models are not capable to represent complex materials (e.g., feathers, hair, cloth, etc.) They also do not capture interreflections, self-shadowing, subsurface scattering. To acquire objects with arbitrary materials we use an image-based modeling and a point-based rendering approach. Image-based representations have the advantage of capturing and representing an object regardless of the complexity of its geometry and appearance. Of course, this advantage comes at the cost of requiring more data. 6 Outline • • • • • Acquisition Model Point-Based Rendering Relighting Reflectance Field Estimation In this talk I will talk about an image-based method to acquire models of complex realworld objects. First I will present the acquisition system, followed by a discussion of the opacity lightfield, which we use to model our objects. Then I will briefly discuss our point-based rendering pipeline with special emphasis on view-dependent shading. After that I will show how opaque objects can be relit using reflectance fields, which allows us to place them into arbitrary new environments. Finally I will discuss a novel method for reflectance field illumination for arbitrary scenes, including reflective, refractive, and transparent objects. 7 Outline ¾Acquisition • Model • Point-Based Rendering • Relighting • Reflectance Field Estimation Let’s start by looking at our image-based acquisition system. 8 Acquisition Systems – I & II V1.0 (July-Oct 2000) V2.0 (Oct-Dec 2000) We have been building image-based acquisition systems for some time. The general idea is to take multiple pictures of the object from different viewpoints. Our first system – shown on the left – consisted of a turntable, a camera, and some static overhead lights. Two light boxes – one opposite the camera and one on the turntable – provided background illumination to make segmentation of the foreground object easier. The second system – shown on the right – added two more cameras to improve the quality of the image-based model. We also mounted the overhead lights on a second turntable so that they can rotate with the object. This ensures that specular highlights remain stationary on the object throughout acquisition. 9 Acquisition System - III This is our third system, which one may call the mother ship of all image-based acquisition systems [Matusik et al. 2002]. Objects are placed onto a plasma display that is mounted on the rotating platform. An array of lights is mounted on an overhead rotating platform. Six video cameras are on the stationary arc and are pointed towards the object. We use the plasma displays for multi-background matting techniques [Smith & Blinn 96] to acquire alpha mattes of the object from multiple viewpoints. We also use them to estimate the light transport through transparent and refractive objects, but more on that later. 10 A lot of headwork… • Building systems is hard, but worthwhile Building these systems is a lot of headwork… But I highly encourage you to try it, because the experience of building systems will focus your thinking. And playing with turntable and cameras is fun… mostly… 11 Acquisition Process Alpha Mattes Surface Lightfield Visual Hull Surface Reflectance Fields The acquisition starts by first acquiring high-quality alpha mattes of the object from different viewpoints 12 Alpha Mattes • Modified multi-background matting [Smith & Blinn 96] with plasma monitors As mentioned before, we use the plasma monitors to display know, random patterns in the background. By comparing the acquired images with images of the known background we can compute the partial coverage of the foreground object with the background matte [Smith & Blinn 96]. The result is a sub-pixel accurate alpha matter, as shown in these examples. After alpha matte acquisition we switch off the plasma displays. 13 Acquisition Process Alpha Mattes Surface Lightfield Visual Hull Surface Reflectance Fields Next we use a threshold to convert the alpha mattes into binary silhouette images. These images are then used to compute an approximate geometry of the object know as the visual hull. 14 Visual Hull • The maximal object consistent with a given set of silhouettes A Visual hull is an approximating geometric shape which is obtained using the silhouettes of an object as seen from a number of views. Each silhouette is extruded creating one cone-like volume that bounds the extent of the object. The intersection of these bounding volumes gives us the visual hull. Note that this method is unable to capture all concavities. 15 Visual Hull Example Here is an animation of the visual hull of a Buzz Lightyear model that was scanned in one of our early systems. Note that the quality of the visual hull geometry is a function of the number of viewpoints / silhouettes. 16 Acquisition Process Alpha Mattes Surface Lightfield Visual Hull Surface Reflectance Fields Now we have two options: We can acquire the surface lightfield of the object – the object under a fixed illumination. That means we rotate the object while the lighting is fixed with respect to the object. 17 Acquisition Process Alpha Mattes Surface Lightfield Visual Hull Surface Reflectance Fields Or we can acquire the surface reflectance field of the object, which is necessary for relightable models. The array of lights is rotated around the object. For each rotation position, each light in the light array is sequentially turned on and an image is captured with each camera. 18 Outline • Acquisition ¾Model • Point-Based Rendering • Relighting • Reflectance Field Estimation From the acquired data we build a model of our objects based on the opacity lightfield, a surface lightfield that includes view-dependent opacity. 19 View-Dependent Textures • Capture concavities, reflections, and transparency with view-dependent textures [Pulli 97, Debevec 98] To add appearance to our model we can use view-dependent (or projective) texture mapping [Debevec et al. 98][Pulli et al. 97]. Here we see our Buzz Lightyear model from different viewpoints. Note that the view-dependent textures capture reflections and transparency and give the illusion of improved concavities. For example, the visual hull does not represent Buzz’s head, although it clearly appears in the texture-mapped model. 20 Surface Lightfield • Parameterize view-dependent textures onto the object surface [Wood et al., 2000] To improve the efficiency for compression and to improve the image quality during view-interpolation we parameterize the view-dependent radiance samples on the visual hull surface of the object. This is called a surface lightfield [Wood et al. 2000]. Note that the visual hull geometry (the black line in the figure) does not correspond to the true geometry of the surface. Nevertheless, the view-dependent surface lightfield is able to capture the actual surface appearance. 21 Opacity Lightfield • Also parameterize view-dependent opacity onto the object surface • 4D function f(u,v,φ,θ) Æ RGBA Opacity (θ,φ) Red = invisible Black = transparent Texture + Opacity θ φ To further improve the appearance, especially for fuzzy or furry objects, we enhance the surface lightfield with view-dependent opacity. We call this new representation the opacity lightfield [Matusik et al. 2002, Vlasic et al. 2003]. The opacity lightfield is a four dimensional function. For each point (u, v) on the object surface it stores radiance and opacity as a 2D function of the viewing direction (phi, theta). In these plots we show the opacity function and the opacity lightfield for one particular surface point. White represents the opaque values, black represents the transparent values, and red represents invisible directions. In general, it is quite difficult to fit a parametric function to the opacity lightfield, although that would certainly be possible for certain materials. It is worth to note that the parameterization surface does not have to be a surface of the visual hull. It can be any surface that contains the actual object. 22 Opacity Lightfield Texture + Opacity Visual Hull + Opacity Lightfield = In summary: Our appearance representation for static light sources is the opacity lightfield, which simply consists of texture + opacity for each acquisition viewpoint. Those textures with alpha are mapped onto the visual hull geometry in a viewdependent fashion. We will discuss the details of rendering of our models later. 23 Example Photo Let’s look at an example. Here we is a photograph of a small bonsai tree. 24 Example Photo Visual Hull This animation shows the visual hull of the tree. Note that the visual hull does not capture any of the fine features, such as leaves and branches. 25 Example Photo Visual Hull Visual Hull + Opacity Here is the visual hull plus view-dependent opacity. It greatly enhances the quality of the object’s silhouette. The background is also visible through the leaves and branches, even though the geometry is exactly the same as on the left. 26 Example Photo Visual Hull Opacity Lightfield Visual Hull + Opacity Finally we map the view-dependent textures onto the surface, too, for the final opacity lightfield model. Note the specular highlights on the base of the tree. 27 Results Video Here is a video of the angel. Look at the feathers, the anisotropic cloth, and the semitransparent cloth on top of that. At some places you will see that the approximate visual hull geometry leads to artifacts. 28 Results Video Here is another video of the bonsai tree, this time with a close-up view. Look at the leaves in the tree and the specular highlights on the vase. 29 Results Video Here is a very specular teapot with intricate texture. You will see some jumping of the highlights in the close-up view. This is due to the limited number of pictures – even though we took 72 x 6 = 432 of them. Very specular objects require a tremendous amount of image data. 30 Results Video Here is a video of a teddy bear. The organic complexity of the silhouette is captured nicely by the opacity lightfield, as is the transparency of the glasses. You will see artifacts in-between the arms of the teddy because of the bad visual hull geometry. 31 Results Video Our acquisition system can also capture organic models, such as living plants, which would be very hard to do with active light or passive systems. 32 Lessons • View-dependent opacity vs. detailed geometry trade-off • Sometimes acquiring the geometry is not possible • Sometimes representing true geometry would be very inefficient • Opacity lightfield stores the “macro” effect There is a logical trade-off between having accurate geometry and the opacity lightfield. If we have good geometry the sampling of the opacity and textures does not need to be very dense. However, if we do not have good geometry we need the viewdependent opacity and textures at a very good sampling rate. There are at least two good reasons why to use the opacity lightfield: Sometimes, geometry is impossible to get. The resolution of the acquisition device – such as range scanner or a camera – may be limited, which means we cannot pick up the details below a certain resolution. And sometimes representing the true geometry would be very inefficient (e.g. for hair). We would simply need too many points or polygons to represent it. The opacity lightfield stores the macro effect, the overall effect of the geometry. 33 Outline • Acquisition • Model ¾Point-Based Rendering • Relighting • Reflectance Field Estimation Let’s now talk about the rendering of our point-based models. In particular we will discuss the view-dependent shading using the opacity lightfield (RGBA) data. 34 Why Points? • Generating a consistent triangle mesh or texture parameterization is time consuming and difficult • Points represent organic models (feathers, tree) much more readily than polygon models What is the connection between our system and point-based graphics? Quite simply, we use a point-based model of the visual hull to store the opacity lightfield at each surface point. The main advantage is that we do not need to generate a consistent triangle mesh or texture parameterization for our models. We avoid difficult issues such as meshstitching, finding a global parameterization of the mesh, texture resampling, texture anti-aliasing, etc. In addition, points are a more natural choice to represent organic models such as feathers and trees, which may have very small features and very complex silhouette lines. 35 Points as Rendering Primitives • Point clouds instead of triangle meshes [Levoy and Whitted 1985] • 2D vector versus pixel graphics triangle mesh (with textures) point cloud The idea of using points instead of triangle meshes and textures has first been proposed by Levoy and Whitted in a pioneering report [Levoy & Whitted 85]. Think of the difference between points and triangles in 3D similar as of the difference between pixels and vector graphics in 2D. Points in 3D are analogous to pixels in 2D, replacing textured triangles or higher order surfaces by zero-dimensional elements. 36 Surface Elements - Surfels • Each point corresponds to a surface element, or surfel, describing the surface in a small neighborhood • Basic surfels: BasicSurfel { position; color; } y position color z x We call a point sample on the object surface a surface element, or surfel. It is similar to a pixel in 2D or a voxel in 3D. As a matter of fact, there is a tight connection between volume rendering and point-based surface rendering (see [Zwicker et al. 2002]). A surfel describes the surface in a (possibly infinitely) small neighborhood. The basic surfel is just a point with a position and a constant color. 37 Surfels • Surfels can be extended by storing additional attributes ExtendedSurfel { position; color; normal; radius; etc... } normal position color radius surfel disc Surfels can of course be extended by storing additional attributes. We also store a normal and a radius with each surfel. This corresponds to having a surfel disc with radius r and normal n. Intuitively, the union of surfel discs covers the object such that there are no holes on the object surface. This will come in handy for reconstructing a continuous image during rendering, as you will see in the talk by Matthias Zwicker. 38 Point Rendering Pipeline Surface samples in image space Input surface samples Shade samples Warp to image space Continuous Reconstruction Filtering, sampling • Simple, pure forward mapping pipeline • Surfels carry all information through the pipeline („surfel stream“) • See Zwicker, Point-Based Rendering The point rendering pipeline processes point data as follows: Input surface samples are first shaded according to the opacity lightfield data. We will discuss this step in more detail next. In the next stage, the points are projected, or warped, to image space. This is analogous to projecting triangle vertices to image space. Next, a continuous surface is reconstructed in image space, which is similar to triangle rasterization. The final stage in the pipeline is filtering and sampling the continuous representation at the output pixel positions, which is is similar to texture filtering and sampling. Matthias Zwicker will talk in much more detail about the reconstruction, filtering, and resampling steps [Zwicker et al. 2001]. Point-based rendering can also be accelerated with commodity graphics hardware, e.g., [Ren et al. 2002]. 39 Point Rendering Pipeline Surface samples in image space Input surface samples Shade samples Warp to image space Continuous Reconstruction Filtering, sampling • Blend colors based on angle between new viewpoint and acquired RGBA images • Linear combination of colors from closest views We will now focus on the pre-surfel shading. The basic shading algorithm blends colors and alphas from the closest images that were acquired from different camera views. The final color per surfel is a linear combination of the colors from the closest views. 40 Color Blending • Unstructured Lumigraph blending [Buehler 2001] • Weights are based on angles between camera vectors and the new viewpoint The formula for this linear combination was developed for Unstructured Lumigraph Rendering by [Buehler et al. 2001]. The blending weights are computed based on the angles between the camera vectors and the new viewpoint. 41 ULR Blending • For each surfel and some view direction d: – Find the k closest camera views – Compute the blending weights ri for each – Normalize to get the final weights wi ri = cos θi − cos θk 1 − cos θi wi = r i k −1 ∑ ri , 1≤ i ≤ k i=1 ULR uses a ranking function based on the angles between original camera views and the desired view. For each surfel v and some viewing direction d, the algorithm finds the k nearest visible views by comparing the angles they subtend with d. Blending weights are computed according to the following formula: ri = cos θi − cos θk 1 − cos θi Here, ri represent relative weights of the closest k-1 views. Those are normalized to sum up to one, resulting in actual weights wi: wi = r i k −1 ∑r i=1 , 1≤ i ≤ k i For more details about this blending formula, including implementation and optimization suggestions, see [Vlasic et al. 2003]. 42 Shading Algorithm • Pre-compute and store visibility vector per surfel • During rendering, project surfel into k closest visible images to get RGBA colors • Blend the RGBA colors using ULR weights ExtendedSurfel { position; visibility[NUM_VIEWS]; normal; radius; etc... } We can further optimize this calculation by pre-computing the visibility of each camera for each surfel by simply projecting it into each camera view. We store the visibility as a binary vector per surfel. For each acquired camera view, the vector stores a 0 (not visible) or 1 (visible). During rendering, we determine the k closest visible camera views for each surfel based on the angles they subtend with the new virtual view. We project the surfel into the k closest images to get RGBA colors for ULR blending. Then we compute the blending weights based on the formula presented on the previous slide and compute the final surfel color. 43 Demo • Real-time for SW-only implementation • No alpha blending • Graphics HW implementation [Vlasic 2003] This demo runs on my laptop in real-time using a software-only implementation. However, we are not doing any alpha blending and the resolution of the textures is limited. Nevertheless, the achievable image quality is very good. We also implemented opacity lightfield rendering on modern graphics hardware. See [Vlasic et al. 2003] for details. 44 Outline • Acquisition • Model • Point-Based Rendering ¾Relighting • Reflectance Field Estimation So far, our models just display the lighting that was used during acquisition. In order to place the models into arbitrary new environments we need to be able to relight them. 45 Surface Reflectance Field • 6D function: R(u, v; θi , Φ i ; θr , Φ r ) • Assumes directional illumination [Debevec 2000] ωi (u,v) ωr To relight our models we acquire the surface reflectance field. The SRF is a 6D function. For each surfel (u,v) and for each incoming light direction at infinity (ωi) and for each outgoing direction (ωo) it stores the ratio of the incoming radiance to the outgoing radiance [Debevec et al. 2000]. It is a simplified form of the BSSRDF for the case of a fixed viewpoint. For opaque objects we capture images of the object and vary the incoming illumination for each turntable position by rotating the overhead light array. We use 4 lights and 15 positions for a total of 60 light configurations per camera view. 46 Relighting – 1st Step • Generate new images for camera views New Illumination Down-sample Surface reflectance field x = Our modified rendering algorithm has two steps. In the first step, we generate the new radiance images for all original camera views given the new illumination environment. Since we acquire the surface reflectance field only for 60 light position we have to severely downsample the new illumination map. 47 Relighting – 2nd step • Interpolate images with ULR blending as before Camera View 1 New View Camera View 2 In the second step we just use standard ULR blending as presented before to compute the color values of each surfel for the new viewpoint. Note that we interpolate both the radiance (color) and the opacity. 48 Results Video In this short video we show some results for opaque 3D models. First, we use directional light sources with random colors to illustrate the relighting. Then we place the 3D model into a synthetic environment where an environment map provides the incoming illumination. We also acquired a short video of a real scene in a local library. The camera was tracked, and we acquired the illumination at the scene using a mirror ball and a camera. Using the camera tracking information and an environment map of the illumination we can then render new views of our objects and composite them into the video. 49 Outline • Acquisition • Model • Point-Based Rendering • Relighting ¾Reflectance Field Estimation We now look at a new method to acquire a surface reflectance field of arbitrary objects, including transparent and refractive ones. Our method uses natural illumination for the estimation of the reflectance field. It works for small objects or large, outdoor scenes. 50 Previous Work • Forward Approaches – Reflectance Fields [Debevec 2000] • Inverse Approaches – Environment Matting [Zongker 99] • Pre-computed Light Transport • We use an inverse approach [Matusik 2004] We can classify the methods for estimating reflectance functions into forward and inverse methods. Most forward methods sample the reflectance functions exhaustively and tabulate the results. For each incident illumination direction they store the reflectance function weights for a fixed observation direction. In practice, only low-resolution incident illumination can be used, since one reflectance table has to be stored per surfel. This is what we used so far. Inverse methods observe an output and compute the probability that it came from a particular region in the incident illumination domain. The incident illumination is typically represented by a bounded region, such as an environment map, which is then modeled as a sum of basis functions. The inverse problem can then be stated as follows: Given an observation (e.g., an image pixel), what are the weights and parameters of the basis functions that best explain the observation? This is what we will discuss now. Note that recent real-time rendering methods use pre-computed light transport (or pre-computed radiance transfer) to increase the realism of their objects. The estimation method for the light transport is similar to what we will discuss now. For a detailed list of references see [Matusik et al. 2004]. 51 Light Transport Model • Incident illumination is 2D parameterizable • Linear “black box” system: B = TL B = relit image (stacked in vector) L = incident light (stacked in vector) T = light transport matrix (reflectance field) First we start with the light transport model that we adopt: We assume that the incident illumination is 2D parameterizable (i.e., can be represented by an environment map). If we consider the problem as a black box linear system, we can write: B = TL where L is the incident illumination (stacked in a vector), B is the resulting relit image (stacked in a vector), and T is the light transport matrix (or reflectance field). Each row in T represents a reflection function Ti, thus T = [T0,…, Tn]. 52 Reflectance Function • Each row Ti of the matrix T is a 4D reflectance function for one output pixel bi Ti (ωi ) = T( x, y; θi , Φ i ) (θi,φi) φi θi Here is an example of a row of the reflectance field matrix T. Each row stores a 4D reflectance function Ti. For each pixel (x,y) and each incoming light direction (ωi) it stores the ratio of the incoming radiance to the outgoing radiance. Note that the outgoing direction is implicitly fixed to be the camera viewpoint. Here we show five samples of this functions for different image points. We draw these samples as a function of the incoming radiance directions phii and thetai. 53 Representation • Approximate Ti as a sum of 2D rectangular kernels Rk,i, each with weight wk,i. Ti ≈ ∑ w k,iRk,i k We approximate Ti as a weighted sum of several 2D rectangular kernels, such that: Ti ≈ ∑ w k,iR k ,i k where wk,i are the weights of each 2D rectangular box Rk,i. The figure shows an example of the axis-aligned 2D kernels. The grayscale color corresponds to the weight magnitude, where white is largest and black is zero. 54 Inverse Estimation • Display input images Lj and record observed images Bj: B j = TL j • Given Lj and bj(x,y), the goal is to estimate T – Weights wk, i – Position and size of the 2D box filters Rk, i • We typically use 200 input images The problem of reflectance field acquisition is now equivalent to estimating the impulse response function (or point spread function) of a linear system with multiple inputs (the incident illumination L) and multiple outputs (the observation B). The estimation process can be described as: B = TL j j illumination (the input images) and Bj is where Lj is a set of different natural incident the resulting set of observed photographs of the scene (the observed images). We use over 200 random input images of indoor and outdoor scenes – such as kitchens, gardens, cities – that are displayed on the monitor. We denote a pixel b(i, j) as pixel i in the j-th observed image. Given L j and b(i, j), the algorithm estimates the weights w k,i, positions, and sizes of the 2D rectangular kernels for each pixel bi in the reflectance field. As we increase the number n of input and observed images, we refine the kernels to improve the estimate of the reflectance field. 55 Algorithm Input and Output • Input: – Images (environment maps) of the input illumination (L1,…,Ln) – Observed images of the scene under input illumination (O1,…,On) • Output: – For each pixel: Weights wk,i, positions and sizes for Rk,i Here again, more explicitly, the input and outputs of our algorithm. 56 Algorithm • Minimize error for each pixel bi: arg min w A i Wi − Bi 2 subject to w ≥ 0 Wi = vector of weights wk,i Bi = vector of observed pixels Ai = matrix of dot products Rk,i • Lj We observe that the problem can be solved independently for each pixel bi. We solve the following optimization problem: arg min w A i Wi − Bi 2 subject to w ≥ 0 Wi is the stacked vector of weights wk,i to be computed (for a single pixel bi). Ai = [Rk,i • Lj ], where Lj is one of the input natural illumination images, and Bi = b i, j We found through experimentation that constraining the kernel weights to be positive is very important for the stability of the solution. This over-constrained system can be efficiently solved using quadratic programming. It estimates the weights Wi that satisfy the system best in the least-squares sense. 57 Kernel Subdivision • Hierarchical quad-tree subdivision of the input image • Typically k=25 kernels per output pixel bi To find the kernel positions and sizes for pixel bi, we use a hierarchical quad-tree subdivision of the input image to find them. We first start with a kernel that occupies the whole image. We split it into two equal size rectangles. Having the option of splitting the kernel horizontally or vertically, we choose the subdivision that yields the lowest error. To compute this error we solve for the kernel weights using the equation in the previous slide. In the second iteration we have four possible splits of the two kernels from the first iteration, and so on. The recursive subdivision stops if K kernels are computed (or K-1 subdivisions have been made).We found that K = 25 yields a good quality-vs.-time tradeoff 58 Kernel Subdivisions Here we show images of the progressive subdivision of the kernels for different scene points indicated by the red arrows. The kernels progress from left to right with 1, 2, 3, 4, 5, 10, 20, and 24 subdivisions. The gray-scale colors of the kernels signify their weight (white is highest). Intuitively, these images show which light from the background is transported towards the pixel. The first row shows how our algorithm quickly (after 10 subdivisions) identifies the small illumination kernel that is refracted by the glass. We identify one pixel (from the 800×600 input image) after 24 quad-tree subdivisions, which is quite remarkable. The second row shows a primarily diffuse point in a hard shadow area. Note how the light transport comes mostly from the right as expected, and how a small reflective component is visible as well. The third row shows a point with subsurface scattering. Light from a large area is scattered towards the viewer. Finally, the last row shows a glossy point. The illumination enters the scene from the left (i.e., where the monitor was placed during acquisition). Note how the light is transported from almost concentric circular areas towards the incoming illumination. 59 Summary • Quadratic programming to estimate weights Wi such that the error is minimized • Quad-tree subdivision of the image to determine the optimal kernels Rk,i Wi initial n +1 Rk,i n initial +1 Wi and Rk,i iterate In summary, our algorithm iteratively proceeds as follows: We first choose the whole image as a kernel and 1 as the initial weight. We then compute the error of this estimation compared with the actual observed images. Assuming the error is too large, we subdivide the kernels and compute the minimum error according to the formula three slides back. This process yields a new set of weights and a set of new kernels. The algorithm proceeds until the error of the estimation is below a certain threshold. 60 Relighting • For each output pixel bi: bi ≈ ∑ w k,i (Rk,i • L i ) k • Weights times the incident light from each kernel • The incident illumination can be stored in a summed-are table to accelerate the computation The pixel value of a single relit output pixel B = [bi] is: bi = Ti • L Note that the pixel value bi is the inner product of two vectors Ti (reflectance function) and L (incident illumination). 61 Results • Single viewpoint, new illumination Estimate Actual As a standard test of the quality of reflectance field estimation, we first compare the actual picture of the scenes under new illumination with the prediction obtained using our estimated reflectance field. In general, our prediction works very well, for glossy, diffuse, and transparent objects. 62 Results Estimate Actual Diffuse elements (e.g., the label on the molasses jar) are reproduced correctly. 63 Results Estimate Actual The Santa model, which is hollow, shows both subsurface scattering and refractions. 64 Moving White Bar • New synthetic illumination Estimate Actual Since our algorithm depends entirely on natural images, the most difficult test for our reflectance field prediction is to use synthetic input illumination with very high frequencies, such as images with white bars on a black background. Here we show our predictions of the scenes with a vertical white bar, 100 pixels wide, sliding across the illumination domain. The glass clearly shows the greasy fingerprints, which corresponds well to reality. However, the movie contains a slight discontinuity due to the vertical subdivision used for the kernels. 65 3D Models • Generate new images for camera views using this technique • Interpolate images with ULR blending • The following results were actually computed using an older technique [Matusik et al. 2002] For 3D models we use the same procedure as before: Generate new, relit images for each of the camera views and interpolate the results using ULR blending for each new viewpoint. The following results were computed using an older technique that uses a combination of low-resolution reflectance fields and environment mattes (see [Matusik et al. 2002]). 66 Results Note the refraction of the checkerboard texture in this scanned wine glass. 67 Transparent Objects Here is our beer glass put into a new, synthetic environment. Note the refraction and the reflections. Note: No beer was harmed to produce this model. We used apple juice, of course. 68 Transparent Objects Note the reflection on the stem of the glass and the refraction. 69 Transparent Objects A crystal sugar bowl. Of course, we do not know what this object really would look like in this environment. 70 Conclusions • Image-based 3D photography is able to capture any type of object • Point-based rendering offers high image-quality for display of complex models • Opacity lightfields provide realistic 3D graphics models • Robust reflectance field estimation allows us to integrated objects into new environments We can draw the following conclusions from our work: Image-based 3D photography is able to capture any type of object, as long as we are willing to acquire enough images. Point-based rendering allows us to display these objects with maximum image quality without compromises due to texture resampling or geometry parameterization. Opacity lightfields offer a simple alternative to having complex geometry, since they allow us to display objects with high complexity using an approximate shape model. Finally, reflectance fields can be estimated robustly, which allows us to relight the objects and place them into arbitrary new environments. 71 Acknowledgements • Colleagues: – MIT: Chris Buehler, Tom Buehler – MERL: Bill Yerazunis, Darren Leigh • Thanks to: – David Tames, Jennifer Roderick Pfister • Papers available at: http://www.merl.com/people/pfister/ We would like to thank these individuals who helped during this research. Please visit my web page to download PDF files of our publications. 72 References • • • • • • • • • • • • • [Debevec et al. 98] Efficient View-Dependent Image-Based Rendering with Projective Texture Mapping. In Proceedings of the 9th Eurographics Workshop on Rendering, 1998. [Levoy et al. 2000] The digital Michelangelo project, SIGGRAPH 2000. [Matusik et al. 2004] Progressively-Refined Reflectance Functions from Natural Illumination, Eurographics Symposium on Rendering 2004. [Matusik et al. 2002] Image-based 3D photography using opacity hulls. SIGGRAPH 2002, July 2002. [Matusik et al. 2002] Acquisition and Rendering of Transparent and Refractive Objects, Thirteenth Eurographics Workshop on Rendering, June 2002. [Pfister et al. 2000] Surfels: Surface elements as rendering primitives, SIGGRAPH 2000. [Pulli et al. 97] View-based rendering: Visualizing real objects from scanned range and color data. In Eurographics Rendering Workshop 1997. [Ren et al. 2002] Object space EWA splatting: A hardware accelerated approach to high quality point rendering, Eurographics 2002. [Smith & Blinn 96] Blue screen matting. Smith, A. R., and Blinn, J. F. 1996. In Computer Graphics, vol. 30 of SIGGRAPH 96 Proceedings. [Vlasic et al. 2003] Opacity Light Fields: Interactive Rendering of Surface Light Fields with View-Dependent Opacity, Proceedings of the Interactive 3D Graphics Symposium 2003, Monterey, April 2003. [Zongker et al. 99] Environment matting and compositing. In Computer Graphics (Aug. 1999), SIGGRAPH 99 Proceedings. [Zwicker et al. 2001] Surface splatting, SIGGRAPH 2001. [Zwicker et al. 2002] EWA Splatting, IEEE TVCG 2002. [Debevec et al. 98] Efficient View-Dependent Image-Based Rendering with Projective Texture Mapping. In Proceedings of the 9th Eurographics Workshop on Rendering, 1998. [Levoy et al. 2000] The digital Michelangelo project, SIGGRAPH 2000. [Matusik et al. 2004] Progressively-Refined Reflectance Functions from Natural Illumination, Eurographics Symposium on Rendering 2004. [Matusik et al. 2002] Image-based 3D photography using opacity hulls. SIGGRAPH 2002, July 2002. [Matusik et al. 2002] Acquisition and Rendering of Transparent and Refractive Objects, Thirteenth Eurographics Workshop on Rendering, June 2002. [Pfister et al. 2000] Surfels: Surface elements as rendering primitives, SIGGRAPH 2000. [Pulli et al. 97] View-based rendering: Visualizing real objects from scanned range and color data. In Eurographics Rendering Workshop 1997. [Ren et al. 2002] Object space EWA splatting: A hardware accelerated approach to high quality point rendering, Eurographics 2002. [Smith & Blinn 96] Blue screen matting. Smith, A. R., and Blinn, J. F. 1996. In Computer Graphics, vol. 30 of SIGGRAPH 96 Proceedings. [Vlasic et al. 2003] Opacity Light Fields: Interactive Rendering of Surface Light Fields with View-Dependent Opacity, Proceedings of the Interactive 3D Graphics Symposium 2003, Monterey, April 2003. [Zongker et al. 99] Environment matting and compositing. In Computer Graphics (Aug. 1999), SIGGRAPH 99 Proceedings. [Zwicker et al. 2001] Surface splatting, SIGGRAPH 2001. [Zwicker et al. 2002] EWA Splatting, IEEE TVCG 2002. 73 Point-based Surface Reps Presenter – Marc Alexa – Discrete Geometric Modeling Group – Darmstadt University of Technology – [email protected] Point-based Computer Graphics Point-based Surface Representations Motivation Overview • Many applications need a definition of surface based on point samples • Introduction & Basics • Fitting Implicit Surfaces • Surfaces from Local Frames – Reduction – Up-sampling – Interrogation (e.g. ray tracing) • Desirable surface properties – Manifold – Smooth – Local (efficient computation) Overview Introduction & Basics • Introduction & Basics • Fitting Implicit Surfaces • Surfaces from Local Frames • Notation, Terms – Regular/Irregular, Approximation/Interpolation, Global/Local • Standard interpolation/approximation techniques – Global: Triangulation, Voronoi-Interpolation, Least Squares (LS), Radial Basis Functions (RBF) – Local: Shepard/Partition of Unity Methods, Moving LS • Problems – Sharp edges, feature size/noise • Functional -> Manifold 1 Introduction & Basics Notation • Notation, Terms • Consider functional (height) data for now • Data points are represented as – Location in parameter space pi – With certain height f i = f (pi ) – Regular/Irregular, Approximation/Interpolation, Global/Local • Standard interpolation/approximation techniques – Global: Triangulation, Voronoi-Interpolation, Least Squares (LS), Radial Basis Functions (RBF) – Local: Shepard/Partition of Unity Methods, Moving LS • Problems – Sharp edges, feature size/noise • Functional -> Manifold • Goal is to approximate f from f i , pi Terms: Regular/Irregular Terms: Approximation/Interpolation • Regular (on a grid) or irregular (scattered) • Neighborhood (topology) is unclear for irregular data • Noisy data ⇒ Approximation • Perfect data ⇒ Interpolation Terms: Global/Local Introduction & Basics • Global approximation • Terms – Regular/Irregular, Approximation/Interpolation, Global/Local • Standard interpolation/approximation techniques • Local approximation – Global: Triangulation, Voronoi-Interpolation, Least Squares (LS), Radial Basis Functions (RBF) – Local: Shepard/Partition of Unity Methods, Moving LS • Problems – Sharp edges, feature size/noise • Locality comes at the expense of fairness • Functional -> Manifold 2 Triangulation Triangulation: Piecewise linear • Exploit the topology in a triangulation (e.g. Delaunay) of the data • Interpolate the data points on the triangles • Barycentric interpolation on simplices (triangles) – Piecewise linear Æ – Piecewise quadratic Æ C1? –… C0 – given point x inside a simplex defined by pi – Compute αi from x = ∑ α ip i i and 1 = ∑ αi i – Then f (x) = ∑ α i f i i Voronoi Interpolation Voronoi Interpolation • compute Voronoi diagram (dual of Delaunay triangulation) • for any point x in space – add x to Voronoi diagram – Voronoi cell τ around x intersects original cells τi of natural neighbors ni – interpolate with f (x) = ∑ λi (x) f i i λi (x) = ∑ λ (x) i i τ ∩ τi τ ⋅ x − pi • Compute Voronoi diagram (dual of Delaunay triangulation) • For any point x in space – Add x to Voronoi diagram – Compute weights from the areas of new cell relative to old cells • Properties – Piecewise cubic – Differentiable, continous derivative τ2 τ3 τ1 n2 τ n1 τ5 x n3 τ4 n5 n4 Voronoi Interpolation Least Squares Properties of Voronoi Interpolation: • linear Precision • local • f(x) ∈ C1 on domain • f(x,x1,...,xn ) is continuous in xi • Fits a primitive to the data • Minimizes squared distances between the pi’s and primitive g g(x) = a + bx + cx 2 min ∑ ( f i − g(pi )) 2 g i 3 Least Squares - Example Least Squares - Example • Primitive is a (univariate) polynomial • Resulting system g(x) = (1, x, x ,...)⋅ c 2 T ( min ∑ f i − (1, pi , pi2 ,...)cT • i ) 2 ⇒ ( 0 = ∑ 2 pij f i − (1, pi , pi2 ,...)cT ) i • Linear system of equations ( ) 0 = ∑ 2 pij f i − (1, pi , pi2 ,...)cT ⇔ i ⎛1 ⎜ p ∑⎜⎜ p2i i i ⎜ ⎝M pi pi2 pi3 ⎛1⎞ pi2 K⎞⎛ c 0 ⎞ ⎟⎜ ⎟ ⎜ ⎟ 3 pi ⎟⎜ c1 ⎟ = 2 f ⎜ pi ⎟ ∑ i ⎟⎜ c 2 ⎟ ⎜ pi2 ⎟ pi4 i ⎟⎜ ⎟ ⎜ ⎟ O⎠⎝ M ⎠ ⎝M⎠ Radial Basis Functions Radial Basis Functions • Represent approximating function as • Solve – Sum of radial functions r – Centered at the data points pi f (x) = ∑ w i r ( pi − x ( f j = ∑ w i r pi − p j ) i ) i to compute weights wi • Linear system of equations ⎛ r (0) r( p 0 − p1 ) r ( p 0 − p 2 ) ⎜ r p − p r (0) r ( p1 − p 2 ) ( ) ⎜ 1 0 ⎜r ( p2 − p 0 ) r( p 2 − p1 ) r (0) ⎜ ⎝ M L⎞⎛ w 0 ⎞ ⎛ f 0 ⎞ ⎟⎜ ⎟ ⎜ ⎟ ⎟⎜ w1 ⎟ = ⎜ f1 ⎟ ⎟⎜ w 2 ⎟ ⎜ f 2 ⎟ ⎟⎜ ⎟ ⎜ ⎟ O⎠⎝ M ⎠ ⎝ M ⎠ Radial Basis Functions Function Spaces! • Solvability depends on radial function • Several choices assure solvability – r (d ) = d 2 log d (thin plate spline) • Monomial, Lagrange, RBF share the same principle: – r (d ) = e−d 2 /h2 (Gaussian) • h is a data parameter • h reflects the feature size or anticipated spacing among points – Choose basis of a function space – Find weight vector for base elements by solving linear system defined by data points – Compute values as linear combinations • Properties – One costly preprocessing step – Simple evaluation of function in any point 4 Function Spaces? Introduction & Basics • Problems • Terms – Many points lead to large linear systems – Evaluation requires global solutions – RBF with compact support • Matrix is sparse • Still: solution depends on every data point, though drop-off is exponential with distance – Local approximation approaches Shepard Interpolation ∑ φ (x) f i i with basis functions • Problems – Sharp edges, feature size/noise • Functional -> Manifold Shepard Interpolation i φ i (x) = • Standard interpolation/approximation techniques – Global: Triangulation, Voronoi-Interpolation, Least Squares (LS), Radial Basis Functions (RBF) – Local: Shepard/Partition of Unity Methods, Moving LS • Solutions • Approach: f(x) = – Regular/Irregular, Approximation/Interpolation, Global/Local ∑ x − xi j x−xj • f(x) is a convex combination of φi, φ i (x ) ≡ 1 because all φ i ∈ [0,1] and • f(x) is contained in the convex hull of data points ∞ • {p i } > 1 ⇒ f(x) ∈ C and ∇ f (pi ) = 0 ∑ −p −p • define f(pi ) = f i = lim f(x) x →p i ÎData points are saddles • global interpolation Îevery f(x) depends on all data points • Only constant precision, i.e. only constant functions are reproduced exactly Shepard Interpolation Partition of Unity Methods Localization: f(x) = ∑ µi (x )φ i (x ) f i • Set i ⎧⎪ 1 − x − p R ν if x − p < R i i) i i • with µi (x) = ⎨( 0 else ⎩⎪ for reasonable Ri and ν >1 Îno constant precision because of possible holes in the data 5 Partition of Unity Methods Partition of Unity Methods • Subdivide domain into cells • Compute local interpolation per cell Partition of Unity Methods Partition of Unity Methods • Blend local interpolations? • Subdivide domain into overlapping cells Partition of Unity Methods Partition of Unity Methods • Compute local interpolations • Blend local interpolations 6 Partition of Unity Methods Partition of Unity Methods • Weights should • Weights should – have the (local) support of the cell – sum up to one everywhere (Shepard weights) – have the (local) support of the cell Moving Least Squares Moving Least Squares • Compute a local LS approximation at x • Weight data points based on distance to x • The set f (x) = g x (x),g x : min ∑ ( f i − g(pi )) θ ( x − pi 2 g ) i is a smooth curve, iff θ is smooth t min ∑ ( f i − g(pi )) θ ( x − pi 2 g(x) = a + bx + cx 2 ) i Moving Least Squares Introduction & Basics • Typical choices for θ: • Terms – – θ (d ) = d θ (d) = e−d – Regular/Irregular, Approximation/Interpolation, Global/Local −r 2 /h2 • Note: θ i = θ ( x − pi • For each x • Standard interpolation/approximation techniques ) is fixed – Standard weighted LS problem – Linear iff corresponding LS is linear – Global: Triangulation, Voronoi-Interpolation, Least Squares (LS), Radial Basis Functions (RBF) – Local: Shepard/Partition of Unity Methods, Moving LS • Problems – Sharp edges, feature size/noise • Functional -> Manifold 7 Typical Problems Functional Æ Manifold • Sharp corners/edges • Standard techniques are applicable if data represents a function • Manifolds are more general • Noise vs. feature size – No parameter domain – No knowledge about neighbors, Delaunay triangulation connects non-neighbors Overview Implicits • Introduction & Basics • Fitting Implicit Surfaces • Surfaces from Local Frames • Each orientable 2-manifold can be embedded in 3-space • Idea: Represent 2-manifold as zero-set of a scalar function in 3-space – Inside: – On the manifold: – Outside: Implicits - Illustration f (x ) < 0 f (x) = 0 f (x ) > 0 Implicits from point samples • Function should be zero in data points – • Image courtesy Greg Turk f (pi ) = 0 • Use standard approximation techniques to find f • Trivial solution: f = 0 • Additional constraints are needed 0 8 Implicits from point samples Implicits from point samples • Constraints define inside and outside • Simple approach (Turk, + O’Brien) + + − − − − + – Sprinkle additional information manually – Make additional information soft constraints • Use normal information + − − + • Normals could be computed from scan • Or, normals have to be estimated + Estimating normals Estimating normals • Normal orientation (Implicits are signed) • General fitting problem min ∑ q − p i ,n θ ( q − p i 2 – Use inside/outside information from scan n =1 n q • Normal direction by fitting a tangent – LS fit to nearest neighbors – Weighted LS fit – MLS fit – Problem is non-linear because n is constrained to unit sphere Estimating normals Implicits from point samples • The constrained minimization problem • Compute non-zero +1 anchors in the distance field +1 • Use normal information directly as +1 constraints min ∑ q − p i ,n θ i 2 n =1 i is solved by the eigenvector corresponding to the smallest eigenvalue of the following covariance matrix ∑ (q − p )⋅(q − p ) θ T i i i i which is constructed as a sum of weighted outer products. ) i f (p i + ni ) = 1 n q +1 +1 +1 +1 +1 +1 +1 +1 +1 9 Computing Implicits Implicits from point samples • Compute non-zero anchors in the distance field • Compute distances at specific points +2 +0.5 +1.5 • Given N points and normals p i , n i and constraints f (p ) = 0,f (c ) = d i +1 −2.5 +1 – Vertices, mid-points, etc. in a spatial subdivision +2.5 +0.5 +2 • Let p i + N = c i • An RBF approximation f (x) = ∑ w iθ ( pi − x i ) i leads to a system of linear equations Computing Implicits Computing Implicits • Practical problems: N > 10000 • Matrix solution becomes difficult • Two solutions • Sparse matrices – Sparse matrices allow iterative solution – Smaller number of RBFs i – Needed: ( ⎛ θ p0 − p1 ⎜ θ (0) ⎜θ ( p1 − p0 ) θ (0) ⎜ ⎜ θ ( p − p 0 ) θ ( p2 − p1 ⎜ M ⎝ ) θ( p − p ) θ ( p1 − p2 ) 0 ) θ (0) d > c → r (d ) = 0, r ' (c) = 0 c 2 ⎞ L⎟ ⎟ ⎟ ⎟ ⎟ O⎠ c – Compactly supported RBFs Computing Implicits RBF Implicits - Results • Smaller number of RBFs • Greedy approach (Carr et al.) • Images courtesy Greg Turk – Start with random small subset – Add RBFs where approximation quality is not sufficient 10 RBF Implicits - Results PuO Implicits • Images courtesy Greg Turk • Construct a spatial subdivision • Compute local distance field approximations +2 +1 +0.5 −2.5 +1.5 +1 – e.g. Quadrics • Blend them with local Shepard weights +2.5 +0.5 +2 Multi-level PuO Implicits PuO Implicits: Sharp features • Subdivide cells based on local error Corner function Local analysis of points and normals Piecewise quadric functions Edge function Standard quadric Multi-level PuO Implicits Multi-level PuO Implicits • Local computations • Aproximation at arbitrary accuracy – Insensitive to number of points • Local adaptation to shape complexity • Sensitive to output complexity 11 Implicits - Conclusions Overview • Scalar field is underconstrained • Introduction & Basics • Fitting Implicit Surfaces • Surfaces from Local Frames – Constraints only define where the field is zero, not where it is non-zero – Additional constraints are needed • Signed fields restrict surfaces to be unbounded – All implicit surfaces define solids Hoppe’s approach Hoppe’s approach - smoother • Use linear distance field per point • Direction fields are interpolated using Voronoi interpolation – Direction is defined by normal • In every point in space use the distance field of the closest point Projection Surface definition • Idea: Map space to surface • Surface is defined as fixpoints of mapping • Projection procedure (Levin) – Local polyonmial approximation • Inspired by differential geometry r – “Implicit” surface definition r’ r r’ – Infinitely smooth & – Manifold surface 12 Local Reference Plane Surface Definition • Constructive definition • Find plane H r = q,n + D 2 – min ∑ q − p i ,n θ ( q − p i Hr – Input point r – Compute a local reference plane Hr=〈q,n〉 – Compute a local polynomial over the plane Gr – Project point r’=Gr(0) – Estimate normal n q, n =1 r Gr Hr n r – Minimize over n for fixed q q q • Eigenvalue problem r −q – Translate q so that n – Along n for fixed direction n r r – q changes Æ the weights change – Only iterative solutions possible q r = q+ r −q n • Effectively changes r − q H n r – Minimize along n for fixed direction n r q • Exploit partial derivative Projecting the Point Spatial data structure • MLS polyonomial over Hr ( ))θ ( q − p ) min ∑ q − pi ,n − G pi H i q • Practical computation r H – LS problem – r’=Gr(0) Hr r Hr n – Non-linear optimization problem ( n ) Local Reference Plane • Minimize independent variables: G ∈Π d /h 2 – Hr is independent of r’s distance – Manifold property • Computing reference plane – θ (d) = e d 2 • h is feature size/ point spacing q Local Reference Plane – Over n for fixed distance – i Weight function based on distance to q, not r n 2 i r r Hr q – Estimate normal • Regular grid based on support of θ – Each point influences only 8 cells • Each cell is an octree r – Distant octree cells are approximated by one point in center of mass Gr 13 Conclusions • Projection-based surface definition – Surface is smooth and manifold – Surface may be bounded – Representation error mainly depends on point density – Adjustable feature size h allows to smooth out noise 14 Point-Based Rendering Matthias Zwicker, Computer Graphics Group, MIT 1 Point-Based Rendering • • • • • • Introduction and motivation Rendering by surface resampling Antialiasing by prefiltering Hardware implementation Applications Conclusions This lecture starts with an introduction and motivation to point-based rendering. Next, point rendering is explained as a resampling process, which builds a conceptual framework for advanced rendering techniques. Basic point rendering algorithms are prone to aliasing artifacts. We present a prefiltering approach to avoid aliasing, which is based on our framework for surface resampling. We then present a hardware implementation of our rendering algorithm and we discuss various applications. We conclude the talk with a summary and directions for future research. 2 Motivation 1 Quake 2, 1998 10k triangles Nvidia, 2002 millions of triangles In recent years, we have witnessed a tremendous development of the computational power of computer graphics hardware. It was not possible to display scenes with more than couple of thousand triangles only a few years ago, which is illustrated on the left with a scene from Quake 2, a state-of-the art 3D game in 1998. Over only a few years, realism of interactive computer graphics has increased drastically. Recent hardware allows interactive rendering of highly complex surfaces, as shown for example in this demo by Nvidia on the right. 3 Motivation 1 • Performance of 3D hardware has exploded (e.g., GeForce FX: up to 338 million vertices per second, GeForce 6: 600 million vertices per second) • Projected triangles are very small (i.e., cover only a few pixels) • Overhead for triangle setup increases (initialization of texture filtering, rasterization) Comparing the performance numbers of the last generations of GPUs, we can say that the performance of these processors has literally exploded. While, for example, the last generation of Nvidia GPUs, the GeForce FX, had a peak vertex rate of 338 million vertices per second, the current generation, the GeForce 6 that has been released this spring, has almost doubled this number to 600 million vertices per second. As a result of this huge processing power, the average projected size of triangles in typical scenes is becoming smaller and smaller. Since the image resolution has not increased significantly, the number of pixels covered by each triangle has decreased significantly. However, this means that the relative overhead for triangle setup, which includes the initialization of rasterization and texture filtering, has increased. It is amortized over fewer and fewer pixels, leading to a higher per pixel rendering cost. 4 Motivation 1 • Simplifying the rendering pipeline by unifying vertex and fragment processing A simpler, more efficient rendering primitive than triangles? Triangle rendering has also become much more flexible and programmable, since modern GPUs allow the application of highly complex per-vertex and per-fragment programs. However, this requires separate functional units for vertex and fragment stages in the rendering pipeline. It would clearly be desirable to simplify the rendering pipeline by unifying the vertex and fragment processing stages. Pointbased rendering has been developed with the goal to provide a rendering primitive that is more efficient than triangles for the highly complex geometry that is common today, and that allows the implementation of efficient and flexible rendering pipelines avoiding redundant functionality. 5 Motivation 2 • Modern 3D scanning devices (e.g., laser range scanners) acquire huge point clouds • Generating consistent triangle meshes is time consuming and difficult A rendering primitive for direct visualization of point clouds, without the need to generate triangle meshes? 4 million pts. [Levoy et al. 2000] While we observe continued advances in the power of rendering and visualization systems, progress in 3D acquisition technology has also increased the complexity of available 3D objects. Modern 3D scanning devices such as laser range scanners acquire huge volumes of point data. Using such devices, the Digital Michelangelo Project by Marc Levoy and his group in Stanford has obtained highly detailed models of various statues by Michelangelo, for example the David statue that is displayed using about 4 million points on the right. To visualize this data using triangle rendering pipelines requires generating consistent triangle meshes, which is a time consuming and difficult process. Therefore, a rendering primitive that allows the direct visualization of unstructured point clouds, without the need to generate triangle meshes as a pre-process, seems very attractive. 6 Points as Rendering Primitives • Point clouds instead of triangle meshes and textures as a rendering primitive [Levoy and Whitted 1985] • 2D vector versus pixel graphics • Points in 3D are analogous to pixels in 2D triangle mesh with textures point cloud, i.e., nonuniform surface samples In this talk, I will present techniques that use points as an alternative to triangles as a fundamental rendering primitive. The idea of using points instead of triangle meshes and textures has first been proposed by Levoy and Whitted in a pioneering report in 1985. Think of the difference between points and triangles in 3D similar as of the difference between pixels and vector graphics in 2D. Points in 3D are analogous to pixels in 2D, replacing textured triangles or higher order surfaces by zero-dimensional elements. 7 Point-Based Surface Representation • Points are nonuniform samples of the surface • The point cloud describes: – 3D geometry of the surface – Surface reflectance properties (e.g., diffuse color, etc.) • Points discretize geometry and appearance at the same rate • There is no additional information, such as – connectivity (i.e., explicit neighborhood information between points) – texture maps, bump maps, etc. Essentially, points are nonuniformly distributed samples of 3D surfaces. A point cloud describes the 3D geometry of the surface, including surface position and normal, and the surface reflectance properties, for example its diffuse color, or even a full BRDF at each point. This means that points are a purely discrete representation of the surface, in fact discretizing the geometry and appearance of the surface at the same rate. Note that in a point-based object representation, there is no additional information such as connectivity, i.e., explicit neighborhood information between points, or texture maps, bump maps, and so on. 8 Model Acquisition • 3D scanning of physical objects [Matusik et al. 2002, Levoy et al. 2000] – See Pfister, acquisition – Direct rendering of acquired point clouds – No mesh reconstruction necessary [Matusik et al. 2002] How are point-based models acquired? Point-based surface representations arise naturally as the raw data produced by 3D scanning devices. Many such systems have been developed recently, making the direct visualization of acquired point clouds desirable and attractive. For example, the acquisition device developed by Matusik et al. is capable of scanning surfaces that are geometrically very complex, such as those shown in the images here. The images are generated using a point rendering algorithm with surface reflectance field shading. Using the points directly as rendering primitives makes mesh reconstruction unnecessary, which would be problematic for the fur of the bear or the feather boa of the angel. 9 Model Acquisition • Sampling synthetic objects – Efficient rendering and simple LOD of complex models (trees, plants) – Dynamic sampling of procedural objects and animated scenes [Zwicker et al. 2001] [Stamminger et al. 2001] [Deussen et al. 2001] Point-based surfaces can also be acquired by sampling synthetic objects, as proposed for example by Pfister et al. and Deussen et al. The goal of these approaches is to render highly complex objects such as trees, plants, or whole ecosystems, as efficiently as possible. In particular, points allow a very simple construction of level-of-detail representations. Points can also be used to dynamically sample and directly visualize procedural objects as described by Stamminger et al. 10 Model Acquisition • Processing and editing of point-sampled geometry (see Pauly, Gross, afternoon session) • Point-based content creation pipeline spectral processing [Pauly, Gross 2002] (Gross, spectral processing) point-based surface editing [Zwicker et al. 2002] (Pauly, Pointshop3D) Shape modeling [Pauly et al. 2003] (Pauly, Pointshop3D) Finally, recent research has extended the scope of point-based surface representations to include processing, editing, and modeling operations. These efforts provide a wide variety of operations to modify point-based surfaces. They will be described in more detail in the talks by Mark Pauly and Markus Gross later in this course. In these techniques, point-based rendering provides the back-end for interactive visualization of the modified objects. Together, the techniques provide a complete point-based content creation pipeline that never requires the reconstruction of a triangle mesh or higher order surface. 11 Rendering by Resampling Input: Resampling: Output: nonuniform set of surface samples rendering algorithm surface samples on a regular image sampling grid • Resampling involves reconstruction, filtering, and sampling • The resampling approach prevents artifacts such as holes and aliasing In this talk, I will introduce point rendering as a resampling process: As an input, we are given an unstructured point cloud, or a nonuniform set of surface samples. The output of the rendering system is again a set of surface samples, but distributed on the regular grid of the output image. The rendering algorithm converts the initial surface samples to new samples at the output positions. Hence rendering is a resampling process that consists of reconstructing a continuous surface from the input samples, filtering the continuous representation, and sampling it, or evaluating it, at the output positions. In this talk I will show how this resampling approach prevents artifacts such as holes or aliasing in point rendering algorithms. 12 Rendering by Resampling • Rendering pipeline Surface samples in image space Input surface samples Shade samples Warp to image space Continuous Reconstruction Filtering, sampling “vertex shading” “vertex projection” “triangle rasterization” “texture filtering” The point rendering pipeline processes point data as follows: Input surface samples are first shaded according to any local illumination model. This stage is comparable to vertex shading in triangle rendering pipelines. In the next stage, the points are projected, or warped, to image space. Again, this is analogous to projecting triangle vertices to image space. Next, a continuous surface is reconstructed in image space, which is similar to triangle rasterization. The final stage in the pipeline is filtering and sampling the continuous representation at the output pixel positions, which is is similar to texture filtering and sampling. 13 Rendering by Resampling • Rendering pipeline Surface samples in image space Input surface samples Shade samples Warp to image space Continuous Reconstruction Filtering, sampling resampling I will now focus on the final three stages of the pipeline, which essentially implement the resampling process. First, I will explain our method for surface reconstruction. 14 Continuous Reconstruction • For uniform samples, use signal processing theory • Reconstruction by convolution with low-pass (reconstruction) filter • Exact reconstruction of band-limited signals using ideal low-pass filters ~ f ( x) = ∑ f (i )δ ( x − i ) i ~ g ( x) = ( f ⊗ h1 2 )( x) = ∑ f (i )h1 2 ( x − i ) i The continuous reconstruction of uniformly sampled functions, or signals, is well understood and described by signal processing theory. Continuous reconstruction is achieved by convolving the sampled signal with a reconstruction or low-pass filter. On the left, the sampled signal tilde(f) is represented as a sum of scaled impulse signals, where f(i) are the samples of the original function. Convolving this with a reconstruction filter h as shown on the right leads to the representation of the reconstructed signal g as a weighted sum of shifted reconstruction filter kernels. One of the major and most practical results of signal processing theory, the socalled sampling theorem, states that band-limited signals can be exactly reconstructed using ideal low-pass filters. 15 Continuous Reconstruction • Signal processing theory not applicable for nonuniform samples • Local weighted average filtering – Normalized sum of local reconstruction kernels ~ f ( x) = ∑ f ( xi )δ ( x − xi ) i ∑ f ( x )r ( x − x ) g ( x) = ∑ r (x − x ) i i i i i i i Unfortunately, signal processing theory is not applicable to nonuniform samples, and it is a non-trivial question what would be the best way to reconstruct a continuous signal from nonuniform samples. Our approach shown on the right is inspired by signal processing theory. We call it local weighted average filtering. The nonuniformly sampled signal is represented by samples f of the original signal at arbitrary positions x_i. We reconstruct a continuous signal g by building a weighted sum of reconstruction kernels r, similar as in the uniform case. However, we use a locally different kernel for each sample position, so this does not correspond to a convolution. Further, we normalize the reconstructed value at each position by the sum of the reconstruction kernels. 16 Nonuniform Reconstruction • Local weighted average filtering – Simple – Efficient – No guarantees about reconstruction error • Normalization division ensures perfect flat field response • Choice of reconstruction kernels based on local sampling density [Zwicker 2003] This local weighted average scheme is very simple and efficient, in particular if the reconstruction kernels have a local support. However, note that it does not guarantee any bounds on the reconstruction error, as signal processing theory does in the uniform case. Further, normalization is a crucial component of the scheme, since it ensures that the reconstruction has a perfect flat field response. This means that reconstructing a constant signal is done exactly and without any error. This is important to avoid visual artifacts. The reconstruction kernels are determined in a pre-process based on the local sampling density. However, a discussion of techniques to do so is outside the scope of this talk. 17 Parameterized Reconstruction • Lower dimensional signal in higher dimensional space common frame? input sample points So far I have been discussing the reconstruction of signals in a functional setting. However, surfaces must be interpreted as lower dimensional signals in a higher dimensional space. In our case, these would be 2D surfaces embedded in 3D Euclidean space. Note that to perform 2D surface reconstruction as described before, we need a global, common 2D coordinate frame. Our solution is to construct such a global frame using a parameterization of the input points. I will illustrate this idea using a 1D curve reconstructed in the 2D plane. First, let us define a local frame for each point, and let us define the reconstruction kernel associated with each point in coordinates of the local frame. Note that the orientation of the local frame and the reconstruction kernel for each point is different. 18 Parameterized Reconstruction • Lower dimensional signal in higher dimensional space local frame reconstruction kernel input sample points parameter domain reconstructed curve x We then use a parameterization to map all the local frames into one global frame of reference. The global frame of reference is also called the global parameter domain. This means that for each local reference frame we define a parameter mapping that converts local coordinates to coordinates of the global parameter domain. I will later explain how this mapping from local to global coordinates is computed for point rendering. Given all the reconstruction kernels in the parameter domain, the reconstruction equation can be evaluated as in the functional setting. However, the reconstructed curve is not a continuous 1D function, but a parameterized 1D curve embedded in the 2D plane. 19 Parameterized Reconstruction • Reconstruction yields a parameterized curve global parameter coordinate sample value reconstruction kernel ∑ f ( x )r (ϕ ( x)) g ( x) = ∑ r (ϕ ( x)) i i i −1 i −1 i i i Parameterization, i.e., mapping to global parameter domain Mathematically, the reconstruction of a parameterized curve can be expressed easily by plugging in the parameter mappings into the reconstruction equation for the functional setting. Here, x denotes the global parameter coordinate, and phi_i^(1)(x) are the mappings from the local frames to the global domain. 20 Parameterized Reconstruction • Surface samples are extended with specification of local reference frame • Usually given by surface normal • Reconstruction of surfaces includes reconstruction of all surface attributes – Color – Position – Normal To apply this scheme to point-sampled surfaces, we need to extend the surface samples by a specification of the local reference frame. Usually, the local reference frame is defined to be perpendicular to the surface normal. Further, note that the reconstruction of surfaces includes the reconstruction of all surface attributes such as its color, position, and normal. All these attributes can be interpolated using the expression shown in the last slide. 21 Parameterized Reconstruction 3D object space 2D parameter domain 2D local reference 2D reconstruction kernel parameterization This slides illustrates the idea of parameterized reconstruction for 2D surfaces in 3D. A 2D local reference plane is defined for each surface sample in 3D, and the 2D reconstruction kernel associated with each sample is defined on this plane. A parameter mapping for each local reference domain transforms the reconstruction kernel to a global 2D parameter domain with coordinates x_0 and x_1. Local weighted average filtering for all surface attributes is then performed in this global parameter domain. 22 Rendering by Resampling • Rendering pipeline Surface samples in image space Input surface samples Shade samples Warp to image space Continuous reconstruction Filtering, sampling Now that I have established a method to reconstruct a continuous surface from the discrete samples, I will explain how it is applied in the point rendering pipeline. The crucial step to do this is the warping or the projection of the input sample points and their attributes to image space. 23 Warping • Use image space as global parameter domain • Warp local reference domains to image space using a 2D to 2D projective mapping • Compute weighted sum of reconstruction kernels in image space The link between parameterized reconstruction described before and point rendering is the idea to use image space as the global parameter domain. The parameter mappings from local to global coordinates are then given by the warp of the local reference domains to image space using a 2D to 2D projective mapping. Reconstruction is performed by computing the weighted sum of reconstruction kernels in image space. 24 Warping The idea of warping the local reference planes to image space and using image space as the global parameter domain is again illustrated on this slide. The reconstruction kernels are initially defined in coordinates u_0, u_1 of the local reference plane shown on the left. The reference planes are embedded in 3D space using an anchor point p and tangent vectors t_u_0 and t_u_1, as shown in the middle. Note that the index i specifying an input point is omitted here to simplify the notation. A 2D to 2D mapping, or a 2D to 2D correspondence, is then established by perspectively projecting these planes to image space with coordinates x_0 and x_1, shown on the right. Hence, the pose of the tangent plane and its perspective projection determine the 2D to 2D parameter mapping x=phi_i(u_i). 25 Warping • The projective mapping from tangent coordinates to image coordinates – Mapping to clip space with coordinates x0’, x1’, x2’, x3’ ⎡ x0 ' (u , v)⎤ ⎢ x ' (u , v) ⎥ ⎢ 1 ⎥ = M ⋅ [t u0 ⎢ x2 ' (u , v)⎥ ⎢ ⎥ ⎣ x3 ' (u , v) ⎦ t u1 ⎡u0 ⎤ p ]⎢⎢ u1 ⎥⎥ ⎢⎣ 1 ⎥⎦ where M is the 4x4 compound modelview-projection matrix – Projection (division by homogeneous coordinate) ⎡ x ' (u , u ) x ' (u , u ) ⎤ x = ϕ (u) = ⎢ 0 0 1 , 1 0 1 ⎥ ⎣ x3 ' (u0 , u1 ) x3 ' (u0 , u1 ) ⎦ The projective mapping from local tangent coordinates to image coordinates is easily expressed using the following matrix notation. First, a point on the tangent plane is mapped to so-called clip space. This is done by transforming a point u_0, u_1 to 3D using the definition of the tangent plane, which is expressed as a matrix vector multiplication here. This point is then mapped to clip space by multiplying it with the 4x4 modelview-projection matrix M, analogous to transforming triangle vertices. Projection to image coordinates is then performed by the division through the homogeneous coordinate x_3. 26 Warping • Reconstruction kernels in image space are also called “footprints” or “splats” * = ∑ f r (ϕ ( j, k )) g ( j, k ) = ∑ r (ϕ ( j, k )) i i −1 i i i −1 i i output pixel coordinates perspective projection of reference domain i The projective 2D to 2D mapping can now be plugged into the reconstruction equation. To simplify explanations, we focus on the reconstruction of the surface color from now on. The color at an output pixel (j,k) is then given by the weighted sum of sample colors f_i multiplied by the value of the warped reconstruction kernels, normalized by the sum of the warped reconstruction kernels at the pixel. The projective mapping phi_i is derived as explained in the slide before. A warped reconstruction kernel in image space multiplied with a sample color is also called a “footprint” or “splat”. 27 Splatting Algorithm for each sample point { shade surface sample; splat = projected reconstruction kernel; rasterize and accumulate splat; } for each output pixel { normalize; } Hence, we also call the resulting rendering algorithm a splatting algorithm. This algorithm proceeds by iterating over all sample points: First, the point is shaded using some local illumination model. Then, the footprint or splat is computed by projecting the reconstruction kernel to image space. The splat is now rasterized and accumulated in the framebuffer. After all samples have been processed, all the values in the framebuffer have to be normalized by the accumulated weights of the warped reconstruction kernels. 28 Normalization without normalization with normalization varying brightness because of irregular point distribution no artifacts I want to point out two important aspects of this splatting algorithm: First, observe that the normalization pass is essential to avoid visual artifacts. Since the input points are nonuniformly distributed in 3D space, it is in general not possible to choose reconstruction kernels that sum up to one everywhere, i.e. that provide a partition of unity. This leads to a varying brightness across the reconstructed surface, as is illustrated in the close-up in the middle. Normalization guarantees a perfect flat field response of the reconstruction scheme, therefore avoiding brightness variations. 29 Visibility • ε-z-buffering surface 1 surface 2 local reference planes z z-buffer pixel accumulate discard splats splats within ε-threshold The second issue is determining the visibility of the splats in image space. In other words, we need to determine which splats belong to the visible surface in a given pixel and therefore should be accumulated, and which ones should be discarded because they belong to an occluded surface. We solve this problem using an extended z-buffering scheme, sometimes also called epsilon-z-buffering. The idea is that in each pixel, all those splats are visible that lie within a range epsilon from the front-most contribution of any splat. All splats that have depth values that are further away than the front-most splat plus the threshold epsilon are considered to belong to a different surface, and they are discarded. 30 Rendering by Resampling • Rendering pipeline Surface samples in image space Input surface samples Shade samples Warp to image space Continuous reconstruction Filtering, sampling Now that we have reconstructed the point-based surface in image space, the final step of the rendering pipeline is to sample the reconstructed signal at the output pixel locations. However to achieve high image quality, it turns out that we need to apply some filtering to the reconstructed signal before sampling it. I will focus on this issue in the next part of the talk. 31 Aliasing • Moiré patterns • Disintegration of textures Unfortunately, rendering using warped reconstruction kernels does not prevent socalled aliasing artifacts. Aliasing artifacts include Moire patterns in images with highly structured patterns, such as the checkerboard texture shown on the right. Moire patterns are low-frequency patterns that appear in sampled versions of highly structured, high frequency signals. An other artifact that we observe is the disintegration of textures with very fine detail, such as the text pattern on the right. These artifacts are analogous to the artifacts produced by texture mapped triangle rendering with poor texture filtering. 32 Antialiasing • Aliasing occurs when sampling a signal that is not band-limited to the Nyquist frequency of the sampling grid • Aliasing is avoided by prefiltering, i.e., band-limiting to the Nyquist frequency, before sampling From signal processing theory we know that aliasing occurs when sampling a signal that is not band-limited to the Nyquist frequency of the sampling grid. The best way to prevent aliasing is to prefilter the signal before sampling, i.e., to band-limit it to the Nyquist frequency of the sampling grid. Prefiltering techniques are common for texture filtering, and the technique that we will describe for point splatting is actually derived from a high quality texture filtering approach. 33 Resampling with Prefiltering source space destination space resampling with prefiltering [Heckbert 89] discrete input discrete output reconstruct sample prefilter warp reconstructed input warped input continuous output Our approach is based on the concept of resampling with prefiltering, which has been introduced to computer graphics by Paul Heckbert in 1989 in the context of texture filtering and image warping. Resampling is the process of taking a discrete input signal, defined in so-called source space, transform it to a destination space, and sample it to a new sampling grid in destination space. To avoid aliasing artifacts, resampling includes the following steps: First, a continuous signal is reconstructed in source space. The continuous signal is then warped to destination space. Next, a prefilter is applied to remove high frequencies from the warped signal, i.e., to band-limit the signal. The band-limited signal is then sampled at the new sampling grid in destination space, avoiding any aliasing artifacts. 34 Resampling with Prefiltering sample value reconstructed input reconstruction kernels position nonuniform spacing Let me describe this in some more detail in the 1D setting: We first reconstruct a signal by building a weighted sum of reconstruction kernels. Note that the kernels are nonuniformly distributed in space. 35 Source Space Destination Space Destination Space Destination Space 4. Sample 2. Warp Resampling with Prefiltering 3. Filter Next, the reconstructed signal is warped to destination space as described before. Here, the warp is simply a 1D dilation of the signal. The warped signal is then lowpass filtered and sampled on the output grid in destination space. 36 Resampling with Prefiltering Source Space Destination Space Destination Space Destination Space warped reconstruction kernel resampling kernel low-pass filter convolution 4. Sample 2. Warp sum of resampling kernel 3. Filter Observe that the warped signal in destination space can again be expressed as a weighted sum of warped reconstruction kernels. Band-limiting the warped signal is implemented as a convolution with a low-pass filter. Since convolution is a linear operation, we can convolve each warped reconstruction kernel with the low-pass kernel individually, instead of applying the low-pass filter to the sum of the reconstruction kernels. We call a warped, low-pass filtered reconstruction kernel a resampling kernel, as shown in the bottom right figure. Hence, the continuous output signal is a weighted sum of resampling kernels, which is evaluated at the output sampling grid. 37 Resampling Filters • Ignoring normalization for now g ( x) = ∑ f i ri (ϕi ( x)) ⊗ h( x) −1 i resampling kernel • The image space resampling kernel combines a warped reconstruction kernel and a low-pass kernel It is straightforward to apply this resampling framework to the reconstruction scheme described earlier in the talk. Let me ignore the normalization division for now. The output signal is given by convolving the reconstructed signal in image space with the low-pass filter. Because of the linearity of the convolution, each warped reconstruction kernel can be convolved separately, yielding a resampling kernel. So the resampling kernel essentially is a combination of a warped reconstruction kernel and a low-pass kernel. 38 Gaussian Resampling Filters • Proposed by [Heckbert 89] g V ( x) = 1 2π V 1/ 2 e 1 − x T V - 1x 2 • Gaussians are closed under – linear mappings: g V ( W -1x) = W g WVWT (x) – convolutions: g V (x) ⊗ g V' (x) = g V + V' (x) While resampling seems to be an elegant approach to do reconstruction and bandlimiting in a unified framework, it is not clear yet how to compute these kernels efficiently. To use resampling filters in practice, Heckbert proposed to use Gaussians as reconstruction and as low-pass filter kernels. The familiar equation for a 2D Gaussian g with a 2x2 variance matrix V is given here. To compute resampling kernels efficiently, we exploit two interesting properties of Gaussian kernels: They are closed under linear mappings and under convolutions. Applying a linear mapping W to a Gaussian with variance matrix V yields an other Gaussian with variance matrix WVW^T. Similarly, convolving a Gaussian with variance matrix V with an other one with variance matrix V’ leads to a new one with variance matrix V+V’. 39 Gaussian Resampling Filters • Locally approximate the projective mapping using its Jacobian – Exact: x = ϕ (u) = [x0 (u0 ,u1 ), x1 (u0 ,u1 )] – Approximation: x ≈ ϕ~ (u) = x 0 + J 0 ⋅ u where ⎡ ∂x0 (u0 , u1 ) ⎢ ∂u 0 J0 = ⎢ ⎢ ∂x1 (u0 , u1 ) ⎢⎣ ∂u0 ∂x0 (u0 , u1 ) ⎤ ⎥ ∂u1 ⎥ ∂x1 (u0 , u1 ) ⎥ ⎥⎦ ( 0, 0 ) ∂u1 Note that since the projective mapping phi from local tangent coordinates to image space is not linear, we can not use it directly to warp the Gaussian reconstruction kernels to image space. Instead, we use a locally linear approximation of this mapping by computing its linear Taylor expansion. This involves the Jacobian J_0, which is the 2x2 matrix consisting of its partial derivatives evaluated at u=0. 40 Gaussian Resampling Filters • Using the affine approximation and the exploiting the properties described before −1 g ( x) = ∑ f i ri (ϕ~i ( x)) ⊗ h( x) i =∑ i 1 f i −1 g J V JT + H (x) i i i Ji Gaussian resampling kernel (EWA resampling kernel) Using the affine approximation and exploiting the properties of Gaussians described before, we can now express the warped and band-limited reconstruction kernel as a single Gaussian kernel. Heckbert has also called this kernel the EWA resampling kernel, EWA standing for elliptical weighted average. The variance matrix of this kernel is composed of the original variance V_i matrix of the reconstruction kernel in tangent space, the Jacobian J_i of the mapping from tangent to image space, and the variance matrix of the low-pass kernel H, which is usually a 2x2 identity matrix. The Gaussian resampling kernel can be computed efficiently, since it only involves one 2x2 matrix inversion and a couple of 2x2 matrix multiplications and additions. 41 Gaussian Resampling Filters warped reconstruction kernel low-pass filter resampling filter minification magnification The effects of Gaussian resampling kernels on the splat shape can be visualized and explained very intuitively. In the left column of this table I depict the shapes of the warped reconstruction kernels in image space. Note that the ellipse corresponds to a contour line of the Gaussian kernel. The shape and size of the reconstruction kernel depends on the projection of the local tangent plane. In case of minification, shown in the top left, the reconstruction kernel will be “smaller” than the pixel spacing, which is indicated on the axes of the figures. In other words, the reconstruction kernel contains higher frequencies than can be captured by the pixel grid, which leads to aliasing artifacts. Convolution with the low pass filter, shown in the middle column, solves this problem. Note that the low-pass filter is independent of the projection, but its size is determined by the spacing of the pixel grid. The resulting resampling kernels are depicted in the right column. Observe that in the case of minification, shown in the top row, the shape of the resampling kernel is largely determined by the low-pass kernel, blurrying the reconstruction kernel and removing its high frequencies. In the middle column, a case of anisotropic minification and magnification is shown. This leads to an anisotropic scaling of the reconstruction kernel as shown on the right. In other words, this filtering technique is equivalent to a so-called anisotropic texture filter. The bottom row illustrates magnification of the reconstruction kernel. In this case, convolution with the lowpass kernel does not change the shape of the kernel significantly, avoiding unnecessary blurrying the output image. 42 Image Quality Comparison • EWA texture filtering [Heckbert 89] surface splatting EWA texture filtering equivalent quality In fact, splatting with Gaussian resampling kernels is mathematically equivalent to EWA texture filtering as originally proposed by Heckbert in 1989. As a result, both approaches provide the same high image quality due to true anisotropic texture filtering. 43 Image Quality Comparison • Ellipse splatting (no low-pass filter) surface splatting splatting reconstruction kernels, aliasing! Here I compare splatting resampling kernels on the left with splatting reconstruction kernels without low-pass filtering on the right. Clearly, this leads to strong aliasing artifacts, visible as Moire patterns on the right. In contrast, those artifacts are completely avoided when using prefiltered resampling kernels. 44 Image Quality Comparison • Trilinear mipmapping surface splatting anisotropic filtering trilinear mipmapping isotropic filtering This is one more comparison of surface splatting on the left with trilinear mipmapping on the right, a popular texture filtering technique. While surface splatting provides anisotropic filtering, trilinear mipmapping is an isotropic filter. However, the amount of filtering required in this example is different in the horizontal and vertical direction. Trilinear mipmapping filters conservatively in both directions, leading to excessive blurriness in the horizontal direction in this example. While the structure of the checkerboard texture is preserved along the horizontal direction on the left, it is completely lost on the right side due to isotropic filtering. 45 Implementation • Software implementation „surface splatting“ [Zwicker et al. 2001] – Edge antialiasing – Order independent transparency • LDC tree data structure [Pfister et al. 2000] – Extension of layered depth images – Hierarchical rendering – Optimized warping The following slides show some results that were generated using a software implementation of our technique. This implementation also features edge antialiasing and order independent transparency to render semi-transparent surface. It is based on the LDC tree data structure, which is an extension of layered depth images. This data structure allows for hierarchical rendering and optimized warping. 46 Results • Scanned head (429‘075 points) This first example is a data set that was acquired using a Cyberware laser range scanner. The scanner obtained roughly 400’000 surface samples, which are visualized directly using our splatting approach. 47 Results • Helicopter (987‘552 points) Here we show a complex object that was sampled from a synthetic model. It contains almost one million points. This animation also illustrates rendering of semitransparent surfaces. 48 Results • Matterhorn (4‘782‘011 points) This is a data set that we acquired from a textured digital terrain model. It contains almost five million points. This animation is to demonstrate the high image quality of our splatting approach. The texture contains very fine detail, which is rendered without flickering or any other aliasing artifacts. 49 Performance • Software implementation [Zwicker et al. 2001] – 1.1 GHz AMD Athlon, 1.5 Gbyte RAM Data Scanned head Helicopter Matterhorn Scanned head #Points 429‘075 987‘552 4‘782‘011 256 x 256 1.3 fps 0.6 fps 0.2 fps Helicopter 512 x512 0.7 fps 0.3 fps 0.1 fps Matterhorn Here is some performance data of the software implementation that was measured on a 1.1 GHz AMD Athlon system. With current systems, rendering performance is increased about a factor of two. We achieved an average of roughly 500’00 splats per second for the models shown before. Note that the performance also depends significantly on the output image resolution. Clearly, this performance is not sufficient for interactive rendering of complex objects. Therefore, we have also attempted to accelerate surface splatting by exploiting current programmable computer graphics hardware, which I will describe in the final part of this talk. 50 Hardware Implementation • Challenges – Extended z-buffering with ε-threshold – Per pixel normalization • Hardware implementation requires a three pass algorithm on current GPUs An implementation of our splatting algorithm on current GPUs faces two challenges. First, the extended z-buffering scheme with an epsilon threshold is not supported directly in current hardware. Second, the per pixel normalization must be done after all splats have been processed and hence it requires an additional pass over the framebuffer. So all implementations on current hardware require three rendering passes, which I will describe in some more detail in the following. 51 First Pass • Draw depth image with a small depth offset ε away from the viewpoint • Perform regular z-buffering (depth tests and updates), no color updates ε surface depth image camera space z In the first pass, we draw a depth image into the framebuffer, which is offset away from the viewpoint by a small depth offset epsilon. We apply regular z-buffering with depth tests and updates, but no color writes. The depth image is visualized on the left. On the right, we show how the depth offset is applied by shifting rendering primitives along viewing rays away from the viewer. 52 Second Pass • Draw colored splats with additive blending enabled • Perform depth tests, but no updates • Accumulate – Weighted colors of visible splats in the color channels – Weights of visible footprint functions in the alpha channel In the second rendering pass, we draw colored with additive alpha blending enabled. In this pass, we perform depth tests, using the depth image computed in the first pass, but no updates. Hence, all splats that lie behind the depth image are discarded. This pass accumulates the weighted colors of the visible splats in the RGB color channels, and the weights of the visible footprint functions in the alpha channel of the framebuffer. 53 Third Pass • Normalization of the color channels by dividing through the alpha channel • Implemented by – using the framebuffer as a texture – drawing a screen filling quad with this texture – performing the normalization in a fragment program The third pass computes the normalization by the accumulated footprint weights by dividing the color channels through the alpha channel. This is implemented by using the framebuffer as a texture. We draw a screen filling quad with this texture, such that there is a one-to-one correspondence between pixels in the framebuffer and texels. Normalization is then performed in a simple fragment program. 54 Drawing Splats • Based OpenGL points (GL_POINT) • Vertex program computes conic matrix for the exponent of the Gaussian ( C = J i Vi J Ti + H )−1 and a bounding box that contains the ellipse xT Cx = r 2 where r is the cutoff radius • The conic matrix C is passed as a texture coordinate to the fragment stage • The bounding box is used to set the OpenGL point size Independent of the three pass rendering approach described before, there are several ways how individual splats can be drawn into the framebuffer. Here we present a method that is based on OpenGL point primitives, which avoids representing each point as a quad or triangle as has been proposed before. Our approach relies heavily on vertex and fragment programs, but it is a quite straightforward implementation of the equations for the Gaussian resampling kernel derived before. In the vertex program, we compute the 2x2 conic matrix for each splat, which is the matrix that appears in the exponent of the Gaussian. This requires the computation of the Jacobian J, the addition of the low-pass kernel H, and the inversion of the resulting matrix. We also compute the size of a bounding box that encloses the ellipse defined by the conic matrix with a certain radius r^2. Here, r is a cutoff radius that clamps the Gaussian kernel to a finite support. The conic matrix C is then passed as a texture coordinate to the fragment stage. Further, the size of the bounding box is used to set the appropriate OpenGL point size. 55 Drawing Splats • The fragment program evaluates d 2 = xT Cx where x are pixel coordinates accessible through the wpos input argument (Cg, NVidia) • Discard fragments if d^2 > r^2 GL_POINT xT Cx = r 2 The fragment program visits each pixel covered by the point primitive and evaluates the quadratic form (x^T)Cx, where x are pixel coordinates that are accessible through the wpos input argument. Note that this currently works only with NVidia cards. We then discard all pixels that yield a value d^2 > r^2, which results in rasterizing an ellipse. Note that by choosing the point size correctly, we guarantee that the ellipse always lies completely within the point primitive. 56 Drawing Splats • Use d^2 to look up the value of the Gaussian in a 1D texture, i.e., 1 2 tex(d 2 ) = e e − d 2 1 − d2 2 d2 Finally, to evaluate the Gaussian kernel at each pixel, we use d^2 to look up the value of the Gaussian in a 1D texture. Note that the texture is indexed directly by d^2 instead of d, avoiding the computation of the square root. 57 Performance • NVidia GeForce FX (NV35) [Zwicker et al. 2004] Image Resolution 512x512 1024x1024 Chameleon Salamander Mio. Splats/Sec. 3.1 2.8 Wasp Fiesta In an implementation on an NVidia GeForce FX, we obtained a rendering performance of about 3 million splats per second. Performance is still dependent on the output image resolution, meaning that the renderer is fragment processing limited for higher output resolutions. 58 Performance • Low-quality hardware accelerated point splatting – up to 50 million points per second on GeForce4 or Radeon 9700 [Dachsbacher et al. 2003] Note that it has been demonstrated that with low-quality hardware accelerated point splatting much higher performance can be obtained. With suitable data structures, Dachsbacher et al. achieved a throughput of up to 50 million points per second on GeForce4 or Radeon 9700 GPUs. These approaches use a single rendering pass and avoid computationally expensive vertex and fragment programs, trading-off performance for high filtering quality provided by surface splatting. 59 Variations of Surface Splatting • Alternative approximation of perspective projection [Räsänen 2002, Zwicker 2004] • Approximation using look-up tables [Botsch et al. 2002] • Efficient hardware implementation [Botsch et al. 2003] • Non-Gaussian kernels [Pajarola et al. 2003] For further reading, let me point out several interesting variations of the surface splatting algorithm that have been presented recently. Rasanen and Zwicker introduced an alternative approximation of the perspective projection that is not based on the Jacobian. Rather, it is based on the exact projective mapping of conic sections using homogeneous coordinates, yielding more accurate splat shapes than the approximation based on the Jacobian. Botsch et al. presented an approximation of surface splatting using pre-computed look-up tables, avoiding the on-the-fly computation of the splat kernels. They also described an efficient hardware implementation that also uses the approach based on OpenGL points described before. Finally, Pajarola et al. proposed to use non-Gaussian kernels to improve filtering quality. 60 Applications of Point Rendering • Direct visualization of point clouds from 3D scanners [Rusinkiewicz et al. 2000, Matusik et al. 2002, Xu et al. 2004] • Real-time 3D reconstruction and rendering for virtual reality applications [Gross et al. 2003] [Rusinkiewicz et al. 2000] [Matusik et al. 2002] [Gross et al. 2003] Point rendering has found a variety of applications. Several researchers have proposed to use point clouds for direct visualization of data acquired by 3D scanners [Rusinkiewicz et al. 2000, Matusik et al. 2002, Xu et al. 2004]. Related to this is using points as a surface representation and rendering primitive in a real time acquisition and rendering environment for virtual reality [Gross et al. 2003]. 61 Applications of Point Rendering • Hybrid point and polygon rendering systems [Chen et al. 2001, Cohen et al. 2001] • Rendering animated scenes [Wand et al. 2002] [Chen et al. 2001] [Wand et al. 2002] Various systems have been proposed to combine traditional rendering primitives with points to get the best of both worlds [Chen et al. 2001, Cohen et al. 2001]. Points have also been proposed to render large animated scenes[Wand et al. 2002]. 62 Applications of Point Rendering • Interactive display of huge meshes [Wand et al. 2001, Deussen et al. 2002] • On the fly sampling and rendering of procedural objects [Stamminger et al. 2001] • Modeling and editing [Zwicker et al. 2002, Pauly et al. 2003] [Deussen et al. 2002] [Stamminger et al. 2001] [Pauly et al. 2003] Points are suitable to build up efficient hierarchies for huge scenes with millions of rendering primitives, which has been explored to interactively display ecosystems with plants and trees [Deussen et al. 2002] or other massive outdoor scenes [Wand et al. 2001]. Finally, points are investigated by several researchers as a primitive for modeling and editing [Stamminger et al. 2001, Zwicker et al. 2002, Pauly et al. 2003]. 63 Conclusions • Point rendering as a resampling process • High quality point splatting with antialiasing using a prefiltering approach • Performance – 3 million high quality splats per second with hardware support [Zwicker et al. 2004] – 1 million points per second in software • Various applications for point splatting have been explored In this talk, I have presented point rendering as a resampling process. Resampling leads to high quality point splatting. It avoids aliasing artifacts using a prefiltering approach and yields image quality equivalent anisotropic texture filtering. The splatting algorithm can be implemented in software or using programmable vertex and fragment stages of modern graphics processors. We have obtained a rendering performance of three million splats per second with hardware support, and roughly one million splats in software. Point rendering has been adopted in various applications, because mesh reconstruction is not necessary and level-of-detail and hierarchy construction is very simple even for highly complex geometries. 64 Future Work • Rendering hardware – Dedicated point rendering hardware – Extension of current hardware • Efficient approximations of exact EWA splatting – Higher performance without loss of image quality • Unifying rendering primitive for all sorts of surface representations – Rendering architecture for on the fly sampling and rendering In the future, we would like to think about dedicated rendering hardware for points, or how to extend existing hardware to more efficiently support points. We would also like to investigate efficient approximations of exact EWA splatting that do not impact image quality, but provide higher rendering performance. Finally, points could be used as a unifying rendering primitive for all sorts of surface representations in a rendering architecture that performs on the fly sampling and rendering. 65 Acknowledgments • Markus Gross, Mark Pauly, CGL at ETH Zurich • Hanspeter Pfister, Jeroen van Baar, MERL Cambridge • Liu Ren, Carnegie Mellon University • Fredo Durand, MIT Graphics Group 66 References • • • • • • • • • • [Levoy and Whitted 1985] The use of points as a display primitive, technical report, University of North Carolina at Chapel Hill, 1985 [Heckbert 1986] Fundamentals of texture mapping and image warping, Master‘s Thesis, 1986 [Grossman and Dally 1998] Point sample rendering, Eurographics workshop on rendering, 1998 [Levoy et al. 2000] The digital Michelangelo project, SIGGRAPH 2000 [Rusinkiewicz et al. 2000] Qsplat, SIGGRAPH 2000 [Pfister et al. 2000] Surfels: Surface elements as rendering primitives, SIGGRAPH 2000 [Zwicker et al. 2001] Surface splatting, SIGGRAPH 2001 [Chen et al. 2001] POP: A Hybrid Point and Polygon Rendering System for Large Data, IEEE Visualization 2001 [Cohen et al. 2001] Hybrid simplification: combining multi-resolution polygon and point rendering, IEEE Visualization 2001 [Wand et al. 2001] The Randomized z-Buffer Algorithm: Interactive Rendering of Highly Complex Scenes, SIGGRAPH 2001 67 References • • • • • • • • [Stamminger et al. 2001] Interactive sampling and rendering for complex and procedural geometry, Rendering Techniques 2001 [Deussen et al. 2002] Interactive Visualization of Complex Plant Ecosystems. IEEE Visualization 2002 [Wand et al. 2002] Multi-Resolution Rendering of Complex Animated Scenes. Eurographics 2002 [Botsch et al. 2002] Efficient high quality rendering of point sampled geometry, Eurographics Workshop on Rendering 2002 [Zwicker et al. 2002] EWA Splatting, IEEE TVCG 2002 [Ren et al. 2002] Object space EWA splatting: A hardware accelerated approach to high quality point rendering, Eurographics 2002 [Räsänen 2002] Surface Splatting: Theory, Extensions and Implementation, MSc thesis 2002 [Matusik et al. 2002] Image-based 3D photography using opacity hulls. SIGGRAPH 2002, July 2002. 68 References • • • • • • • • [Zwicker et al. 2002] Pointshop 3D: An interactive system for editing pointsampled surfaces. SIGGRAPH 2002 [Pauly et al. 2003] Shape modeling with point-sampled geometry. Siggraph 2003 [Pajarola et al. 2003] Object-Space Point Blending and Splatting. In Sketches & Applications Catalog SIGGRAPH 2003 [Botsch et al. 2003] High-Quality Point-Based Rendering on Modern GPUs. Pacific Graphics 2003 [Gross et al. 2003] blue-c: A Spatially Immersive Display and 3D Video Portal for Telepresence. SIGGRAPH 2003 [Dachsbacher et al. 2003] Sequential point trees. SIGGRAPH 2003 [Zwicker et al. 2004] Perspective accurate splatting. Graphics Interface 2004 [Xu et al. 2004] Stylized rendering of 3D scanned realworld environments. NPAR 2004 69 Efficient Data Structures Marc Stamminger, University of Erlangen-Nuremberg 1 Introduction • point rendering – how to adapt point densities ? • for a given viewing position, how can we get n points that suffice for that viewer ? – how to render the points ? • given n points, how can we render an image from them ? When using points for efficient rendering, we have to consider two issues. First, how to adapt the point densities for a given camera to avoid holes but not to waste resources. Second, how to render the selected points. We can use software or hardware based approaches for point rendering or splatting. When using hardware acceleration, it is very important to restrict data transfer between main memory and the GPU to a minimum. 2 Introduction • how to render the points ? – project point to pixel, set pixel color – hardware acceleration • >80 mio. points per second (Radeon 9700) • very fast, but bad support for splatting The simplest way of rendering the points is to project them to the screen and set the corresponding pixel. Graphics hardware can do this using the point primitive very efficiently, at least as long as we don‘t want to have sophisticated filtering and splatting techniques. Current graphics hardware can render almost 100 million points per second. 3 Introduction • for ( i = 0; i < N; i++ ) renderPointWithNormalAndColor (x[i],y[i],z[i],nx[i],ny[i],nz[i],…); → ~10 mio points per second immediate mode • for ( i = 0; i < N; i++ ) renderPoint(x[i],y[i],z[i]); → ~20 mio points per second • float *p = {...} renderPoints(p,N); → >80 mio points per second → best performance with sequential processing of large chunks ! In this talk, we want to use hardware rendering for the points. This is by far the fastest way to render points, and it can be foreseen that future hardware will support high-quality splatting techniques better. For hardware rendering, it is vital to think about the geometry submission to the GPU. When using e.g. OpenGLs immediate mode, we have a tremendous overhead of functions calls and also memory bandwith problems. The more attributes we associate with a single point sample, the worse it gets. When rendering point primitives with a normal and color value, we can achieve a point rate of approximately 10 million splats per second. Using coordinates only, we achieve about 20 million points per second, so memory bandwith is the bottleneck. The shown numbers are not really measured, they only give an idea of the magnitude. The most performant rendering is achieved by processing large chunks of data at once, in particular, if this data is already stored in video memory. 4 Intro • in this talk: two approaches – eco system rendering with points, lines, and triangles – sequential point trees In this talk, we will present two techniques for point rendering on graphics hardware. The first one is optimal for objects like plants, with complex, unconnected geometry. The second one is better suited for complex, but connected geometry. 5 Eco System Rendering • point list – array of random points • for every frame – compute n, render first n 100,000 10,000 1,000 Let‘s start with the rendering of complex objects such as plants. Our idea is simple: we precompute a list of random points on the object and store them together with their normal and color in an array. Because the points are randomly chosen, we can get n random points on the object by simply chosing the first n points. So, for every frame we compute an appropriate number of points n and render the first n points from the array. According to the observations before, this rendering of a continuous segment of the array can be done at maximum speed by the GPU. 6 Eco System Rendering • pros + maximum efficiency + simple tradeoff speed-quality • cons - randomness generates random holes 100,000 10,000 1,000 By increasing the point size, less points are required and speed can be easily balanced versus quality. The disadvantage is that the randomness generates holes. Avoiding these holes requires to increase the number of points enormously, which results in a breakdown of speed. 7 Eco System Rendering • ideal for plants – very high complexity – artifacts from holes acceptable – other LOD-methods fail for complex topology • bad for smooth surfaces – holes are unacceptable – triangles more appropriate – better LODs available However, for plants with many small surfaces (leaves), holes are not noticable and can be tolerated. Interestingly, such unconnected topologies like trees are particularily difficult for most other level of detail methods. For smooth, large, closed surfaces, this approach is hardly useful and other techniques are better suited. 8 Eco System Rendering • same idea works with lines points lines polygons A similar idea works with lines. We can replace long, thin geometry in the distance by lines, e.g. gras blades. 9 Eco System Rendering • line rendering 10 Eco System Rendering • improved level-of-detail triangles points lines triangles points lines triangles points lines triangles points lines We start our approach with a triangle model. Part of the triangles can be replaced by lines, the rest by a set of points. These sets of lines and points are precomputed or generated during the evaluation of the procedural plant model. When we are close to the object, the full triangle model is rendered. When the object is moved away, at some point the triangles that can be represented by the lines are replaced by the line set. Furthermore, we start replacing more and more triangle by their corresponding points. By this, we avoid that the triangles are immediately replaced by points and we thus avoid the popping artifacts. When all triangles are replaced by points, we can reduce the point number for distant objects. 11 Eco System Rendering 12 Sequential Point Trees • sequential point trees – adaptive point selection and – sequential processing • rendering – pre-computed point and attribute list – render continuous segments only rendered segment point list The second technique presented in this talks are sequential point trees. SPTs are a data structure that allows us to render levels of detail also of closed, smooth surfaces without holes. SPTs can also be processed almost completely on the GPU with very high point throughput. As before, we precompute an array of points on the object and only process a continuous part of this list. 13 Hierarchical Processing • Q-Splat – Rusinkiewicz et al., Siggraph 2000 – hierarchical point rendering based on bounding sphere hierarchy R R R R R R R © Rusinkiewicz Our method is based on the Q-Splat as introduced by Rusinkiewicz et al at Siggraph 2000. The Q-Splat does hierarchical point rendering using a bounding sphere hierarchy. Every node represents a part of the objects surface with a position, a radius and a normal. During rendering this hierarchy is traversed by the CPU and the tree data structure is used for frustum culling, backface culling and level of detail selection. 14 Q-Splat Rendering • recursive rendering R R R R image size > 1 pixel → traverse children R R R The traversal of the sphere hierarchy always begins at the root node. The bounding sphere of the currently considered node is projected onto the image plane. If its size is larger than a user defined threshold, e.g. 1 pixel, the image error is too large and the child nodes are traversed. 15 Q-Splat Rendering image • recursive rendering R R R R image size > 1 pixel → traverse children R R R As we can see in this illustration, the image size of the node in the next hierarchy level is still slighty above our threshold. So we skip this node, too… 16 Q-Splat Rendering image • recursive rendering R R R R image size < 1 pixel → render disk R R R Finally, the image size of this node is clearly below our threshold and it is rendered. Typically we render disks or simple opaque squares instead of spheres. 17 Hierarchical Processing • • • • • straightforward, but drawbacks not continuously stored in array not sequential traversal by CPU, rendering by GPU CPU is bottleneck → sequential version ? This kind of data processing is very simple and straightforward, but unfortunately it has some major drawbacks for use with nowadays graphics hardware. The rendered points are not continuously stored in an array and are not processed sequentially. The CPU traverses the tree and invokes the rendering of independent nodes. This ends up in a bottleneck and the power of the GPU is not fully used. So what we would like to have, is a sequential version of the adaptive point selection. So we can offload more work to the GPU and bypass the bottlenecks. 18 Sequential Point Trees • Q-Splat: render node if – image size ≤ threshold and – image size of parent > threshold – image size = radius / view distance • store with node n.dmin = n.radius / threshold • render node n if – view distance(n) ≥ n.dmin and – view distance(parent) < parent.dmin To develop a sequential version of the traversal, let‘s have a look what happens with a single node of the Q-Splat tree. A node is rendered only if it‘s image size is below a certain threshold AND the image size of it‘s parent node is above the threshold. This second criterion is false, if a predecessor is already small enough to be rendered, so the hierarchical traversal of the sub-tree would be skipped. The image size of a node can be computed very easily. It is simply the radius of the sphere over the view distance. Instead of regarding the image size of a node, we use another equivalent measure. For each node we can derive a minimum distance d_min, which equals to its radius over the threshold. If the node is closer to the viewer than d_min, its image size is too large and we don‘t want to render it. Then we can redraft our criterion: a node is rendered, if it‘s view distance is above or equal to d_min, and the view distance of its parent node is smaller than the parents‘ d_min value. 19 Sequential Point Trees • if we assume view distance(n) ≈ view distance(parent) • and set: n.dmax = parent.dmin • criterion for node n: n.dmin ≤ view distance(n) < n.dmax image size small enough parent image size too big For now let‘s assume, that the view distances of child nodes and their parent nodes are roughly equal. In addition to the d_min value, we store a d_max value for each node, which is simply the d_min value of its parent node. In this way, we can get rid of the hierarchical data structure and define a two-sided test as rendering criterion for a single node. The d_min boundary guarantees, that the image size of the node is small enough, the d_max boundary discards this node if its parent node should be rendered instead. 20 Sequential Point Trees • example tree [dmin,dmax] This slide shows an example tree with 13 nodes. Each node brings along ist d_min d_max interval as shown. 21 Sequential Point Trees • loop over all tree nodes • first sequential version foreach tree node n d = view distance(n); if ( n.dmin ≤ d < n.dmax ) renderPoint(n); With this criterion we can formulate the rendering as a simple loop over all tree nodes. For each node, we calculate its view distance and render the node, if it passes the interval test. 22 Sequential Point Trees • account for view distance(n) ≠ view distance(parent): – n.dmax = parent.dmin + |parent-n| – partially parent and some children selected – overdraw, but no visible artifacts In general the view distance of a node and the view distance of its parent node are not equal. So our previous assumption can cause rendering errors. To prevent this, we introduce an interval overlap and increase a node‘s d_max value by the distance between the child and the parent node. This overlap ensures that no holes appear, but it also means that for some nodes both, the node and some of its childrens are selected. This results in overdraw and very slightly reduced performance, but we did not experience visible artefacts from it. 23 Sequential Point Trees • how enumerate nodes? • sort by dmax [dmin,dmax] This slide shows an example tree. Each node brings along its d_min d_max interval. To get a sequential version of the tree nodes, we need to enumerate the nodes so that selected list entries for particular views are preferably densely clustered in a segment of the list. The solution we chose is, to sort the nodes by descending d_max values. If the view distance to the object is greater than 10, we only render node A. If we move closer to the object, let‘s say the distance is now 8, then the nodes B, C and D are rendered. When moving even closer, we need to some of the blue nodes to represend sufficient detail, for example the KLM nodes instead if the D node. 24 Sequential Point Trees • dlow: lower bound on view distance(n) dlow During rendering, we calculate a lower bound of the node‘s view distances for the whole object. This can be done very easily, for example by using bounding volumes. This value is denoted by d_low. 25 Sequential Point Trees • cull nodes with dmax less than dlow • interval test for remaining prefix – view distance is not constant for all nodes ! dlow=8 dlow=4.5 dlow=3.5 All nodes, which have a d_max value smaller than our lower bound can be immediatly skipped. These are the nodes at the tail of the list. Only the nodes in a prefix of the list MAY pass the interval test. On this slide, you can see the prefixes resulting from different three different d_low values for our example tree. Since the view distance is not constant for all nodes and the prefix contains nodes not suitable for the current view, we still need culling for individual nodes by the GPU. 26 Sequential Point Trees • culling by GPU necessary, because d is not constant over object Up to now, we assumed that the view distance is the same for all points. In this case, every view distance d defines front of points in the hierarchy that is to be rendered for this view distance. The SPT is sorted such that this point front and all points above them are the prefix of the SPT array. In fact, the view distance varies over the object. By this, we have an additional (green) interval, where we between the point front for the minimum and maximum view distance. For points inside this interval, we have to evaluate the view distance and make the test again. The point front to be rendered is in this interval region. 27 Sequential Point Trees • CPU does per frame: – compute dlow – binary search: last node with dmax ≥ dlow – send draw command for prefix to GPU – store SPT in video memory • GPU then does for every node n – compute d = view distance(n) – if n.dmin ≤ d < n.dmax • render node This slide summarizes all tasks needed for the rendering. For each frame, the CPU computes the lower bound of node distances. Using a binary search, the prefix can be determined very quickly and the CPU can send the draw command to the GPU. The sequential point tree is preferably stored in video memory as a point primitive list, where the d_min d_max interval is stored as an attribute for each point. The GPU, using vertex programs or vertex shaders, computes the view distance for each node. The view distance is simply the z value of the clip coordinates of the point. If a node passes the test, it is rendered, otherwise, the point primitive is culled by moving it to infinity. 28 Sequential Point Trees • CPU prefix selection (coarse granularity) • GPU point selection (fine granularity) sequential point tree prefix with dmax ≥ dlow culled because d > dmax rendered culled because d < dmin So the CPU does fast coarse granularity culling and the GPU selects single points for rendering. This slide shows the whole sequential point tree and the selected prefix. Nodes at the beginning of the prefix may be culled because they are too far up the hierachy and thus too large. They are outside the d_min boundary. They are not really critical, because in a tree with branching factor four, only ¾ of the nodes are leaves. We could make an equivalent search for a left boundary, however, the list is not sorted for d_min, so this search is costly and we will not find a tight bound. At the end of the prefix nodes are culled, because they are to far down the hierarchy. They are only within the prefix, because the prefix length is determined conservatively. Both tests have to be done by the GPU as they require the exact view distance of each node. 29 Sequential Point Trees • example 30 Sequential Point Trees • Results – culling by GPU: only 10 - 40% – on a 2.4 GHz Pentium with Radeon 9700: – CPU-Load < 20% (usually much less) – > 50 Mio points after culling Before we‘re going more into detail of the error measurements and improvements, let‘s have a look at the results. The percentage of points culled by the GPU depends on the variation of the view distances over the object. In typical examples this fraction is 10 to 40% of the prefix. On a 2,4 GHz Pentium 4 processor with a radeon 9700, the CPU load is always below 20%. Usually it is much less. When rendering a single sequential point tree, the CPU load is even below 1%. The effective point rate, that means the number of points which pass the interval test and are rendered, is more than 50 million points per second. 31 Sequential Point Trees • geometric error measures – perpendicular error ep – tangential error et Let‘s have a look at further improvements… Every node in the hierarchy can be approximated by a disk with a certain center, normal and diameter. To introduce a more sophisticated error measure, we distinguish two different types of geometric errors: 1.) The first one is the perpendicular error. It is the minimum distance between two planes parallel to a parent disk, that encloses all child disks. It captures the fact, that errors along edges are less acceptable. 2.) Second, we introduce the tangential error. It looks at the projection of the child disks onto the parent disk. It measures if the parent disk covers and unnecessary large area, resulting in typical errors at surface edges. We measure it by fitting two parallel slabs of varying orientation around the projected child disks. The tangential error is defined as the difference of the disk diameter and the width of the tightest slab. 32 Sequential Point Trees • error projected into image – ep scales with sin(α)/d et scales with cos(α)/d α=angle(normal, view direction) • single geometric error • eg scales with 1/d When the geometric errors are projected into the image, then the image space counterpart of the perpendicular error is propotional to the sine of alpha and 1 over d. The tangential error in image space is proportional with the cosine of alpha and 1 over d. Where alpha is the angle between the disk normal and the view direction. To fit these error measures into the sequential point trees, it‘s important to obtain a single geometric error e_g, as shown on the slide, which scales with 1 over the view distance. 33 Sequential Point Trees • less, but larger splats in less detailed regions Results: In flat parts of the surface, less but larger splats are rendered, here shown in yellow and green. More detailed parts require more point samples shown in red. The bottom right bar, shows the sequential point tree and the prefix sent to the GPU in red and green. If the object moves away, the prefix becomes smaller. The blue fraction of the list is culled by the CPU. 34 Sequential Point Trees • what about colors ? • in flat textured regions – washing out texture detail • add texture criterion – if significant color variation in child nodes: increase error to node diameter Sequential Point Trees can also contain color information. Every leaf of the point hierarchy represents a part of a surface and an average color can be assigned. For inner nodes of the hierarchy the color values of the child nodes are averaged. With color averaging, we have to reconsider our error measure. In flat regions with small geometric error, large splats are rendered and the texture detail is washed out. To avoid this, we increase a node‘s error to the points diameter, if the color of child nodes varies significantly. 35 Sequential Point Trees • combine errors • etex = diameter if texture varies 0 else • ecom = max( etex, eg ) image error = ecom / view distance n.dmin = n.ecom / threshold Both errors scale with 1 over d, and can be combined to a single error measure to fit into the sequential point tree concept. The texture error e_tex equals zero or the node`s diameter, depending on the color variation of the child nodes. The combined error e_com is then the maximum of e_tex and the geometric error e_g. The image error is then e_com over the view distance. And a node`s d_min value for the interval test is then simply e_com over the threshold. 36 Sequential Point Trees • enforce small splats, reduce blurring to threshold • point densities adapt to texture detail This enforces small splats and the blurring is reduced to the error threshold. With this measure, point densities adapt to texture detail as you can see on the left image. 37 Sequential Point Trees • hybrid point-polygon rendering • don‘t render large triangles with points Sequential point trees can be extended to hybrid point-polygon rendering. In these images, we can see a very artificial example of a pillar. On the left image, it is smooth shaded, on the right hand side, the parts shown in red are rendered using points, the parts with less curvature are rendered as triangles, shown in gray. 38 Sequential Point Trees • combine with polygonal rendering – for every triangle • compute dmax = longest side / threshold • remove all points from triangle with smaller dmax – sort triangles for dmax – during rendering • send triangles with dmax < dlow to GPU • on the GPU (vertex program) – test d < dmax – cull by alpha-test – border case: differently classified vertices partially rendered triangles Rendering a triangle is reasonable as long as its longest side has an image size above our error threshold. Thus we can compute a d_max value for each triangle. We can remove all point samples of a triangle which have a smaller d_max value than the triangle. Because it will always pay off to render the triangle instead of these points. Like the point hierarchy, we sort all triangles according to their d_max value. During rendering, the required prefix of the triangle list is determind in the same way as for the point list. A vertex program evaluates the d_max condition for every vertex and puts the result into the alpha value of the vertex. Culling is then done by an alpha-test. By this, triangles with differently classified vertices are rendered partially. Since this is a border case, the corresponding point samples are also rendered and resulting holes are automatically filled. 39 Sequential Point Trees • pros – very simple! – continuous level of detail – mostly on the GPU – GPU runs at maximum efficiency • cons – no hierarchical view frustum culling – currently: bad splatting support by GPU The sequential point trees are very simple and easy to implement and provide continuous level of detail rendering. Since almost all work is moved to the GPU, the CPU load is very low and can be used for other tasks. If the point list is stored in video memory, the GPU runs at maximum efficiency and the rendering speed is limited by the geometry processing, when using rather simple splatting techniques. The sequential point trees do not allow hierarchical frustum culling within an object. Objects can only be culled on the whole or have to be split into several point trees. And this is, what we did for backface culling: We split up all points samples into several normal clusters and only render front facing clusters. This requires slightly more computation, but skips about 50 percent of the point samples. Unfortunately contemporary graphic hardware has bad support for splatting techniques. 40 Sequential Point Trees • video: sculpture garden In this video, we can see a real-life example captured from a Radeon 9700. The sculptures and the trees are rendered using sequential point trees, the other objects in the scene and the sky is rendered as textured triangles. The red-blue bar shows the very low CPU load required for this scene. The flickering seen at the sculptures results from the simplistic splatting technique we used for this demo. 41 Questions ? …thank you for your attention ! contact: [email protected] url: www9.cs.fau.de/Research/Rendering 42 Overview • • • • Spectral Processing of Point-Sampled Geometry Introduction Fourier transform Spectral processing pipeline Applications – Spectral filtering – Adaptive subsampling • Summary Markus Gross Introduction Introduction • Idea: Extend the Fourier transform to manifold geometry • Applications: – Spectral filtering: • Noise removal • Microstructure analysis • Enhancement – Adaptive resampling: Ö Spectral representation of point-based objects • Complexity reduction Ö Powerful methods for digital geometry processing • Continuous LOD Fourier Transform Fourier Transform • Requirements: • 1D example: N X n = ∑ xk e k =1 output signal – Fourier transform defined on Euclidean domain nk − j 2π N input signal • Benefits: – Sound concept of frequency – Extensive theory – Fast algorithms Ö we need a global parameterization spectral basis function – Basis functions are eigenfunctions of Laplacian operator Ö requires regular sampling pattern so that basis functions can be expressed in analytical form (fast evaluation) • Limitations: – Basis functions are globally defined Ö Lack of local control 1 Approach Spectral Pipeline • Split model into patches that: – are parameterized over the unit-square Ö mapping must be continuous and should minimize distortion – are re-sampled onto a regular grid Ö adjust sampling rate to minimize information loss – provide sufficient granularity for intended application (local analysis) Ö process each patch individually and blend processed patches Patch Layout Creation Clustering Patch Layout Creation Ö Optimization • Iterative, local optimization method • Merge patches according to quality metric: Φ = Φ S ⋅ Φ NC ⋅ Φ B ⋅ Φ Reg ΦS Samples Ö Clusters Ö Patches Ö patch Size Φ NCÖ curvature ΦB Ö patch boundary Φ RegÖ spring energy regularization Patch Layout Creation Patch Resampling • Parameterize patches by orthogonal projection onto base plane • Bound normal cone to control distortion of mapping using smallest enclosing sphere • Patches are irregularly sampled 2 Patch Resampling Spectral Analysis • Resample patch onto regular grid using hierarchical push-pull filter (scattered data approximation) • 2D discrete Fourier transform (DFT) Ö Direct manipulation of spectral coefficients • Filtering as convolution: F ( x ⊗ y ) = F ( x) ⋅ F ( y ) Ö Convolution: O(N2) Ö multiplication: O(N) • Inverse Fourier transform Ö Filtered Spectral Filters Spectral Filters • Smoothing filters ideal low-pass patch surface • Microstructure analysis and enhancement Gaussian low-pass original transfer function: spectral domain transfer function: spatial domain Spectral Resampling Reconstruction • Low-pass filtering • Filtering can lead to discontinuities at patch boundaries Ö Band-limitation Ö Create patch overlap, blend adjacent patches • Regular Resampling region of overlap Sampling rates Ö Optimal sampling rate (sampling theorem) Ö Error control (Parseval’s theorem) Power Spectrum Point positions Normals 3 Reconstruction Timings • Blending the sampling rate blended sampling rate in region of patch overlap discretized sampling rate on regular grid Time pre-computed sampling patterns Applications • Surface Restoration • Interactive filtering Gaussian low-pass Wiener filter 9% Patch Merging 38% SDA 23% Analysis 4% Reconstruction 26% Applications Original Clustering Patch layout Applications Applications - Watermarking • Adaptive Subsampling • Spectral embedding and readout Input model 4,128,614 pts. = 100% 287,163 pts. = 6.9% Patching & spectral decomposition Combined attack (noise, clip, affine transform) Cotting, Weyrich, Pauly, Gross, SMI 04 4 Extensions Filtering • Scale spaces: levels of smoothness Pauly, Kobbelt, Gross, ACM TOG, to appear • Sequence of progressively smoothed models • Partitioning into spectral bands Summary Reference • Versatile spectral decomposition of point-based models • Pauly, Gross: Spectral Processing of Point-sampled Geometry, SIGGRAPH 2001 • Effective filtering and spectral analysis • Adaptive resampling • Efficient processing of large point-sampled models 5 Overview • • • • • Introduction Local surface analysis Simplification methods Error measurement Comparison Surface Simplification Mark Pauly Stanford University Point-Based Computer Graphics Surface Simplification Mark Pauly Introduction Introduction • Point-based models are often sampled very densely • Many applications require coarser approximations, e.g. for efficient • Example: Level-of-detail (LOD) rendering – Storage – Transmission – Processing – Rendering 10k Ö We need simplification methods for reducing the complexity of point-based surfaces Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics 20k 60k 200k Surface Simplification 2000k Mark Pauly Introduction Local Surface Analysis • We transfer different simplification methods from triangle meshes to point clouds: • Cloud of point samples describes underlying (manifold) surface • Hierarchical clustering • Iterative simplification • Particle simulation • We need: – Mechanisms for locally approximating the surface Ö MLS approach – Fast estimation of tangent plane and curvature Ö principal component analysis of local neighborhood • Depending on the intended use, each method has its pros and cons (see comparison) Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 1 Neighborhood Neighborhood • No explicit connectivity between samples (as with triangle meshes) • K-nearest neighbors • Replace geodesic proximity with spatial proximity (requires sufficiently high sampling density!) • Compute neighborhood according to Euclidean distance – Can be quickly computed using spatial datastructures (e.g. kd-tree, octree, bsp-tree) – Requires isotropic point distribution Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly Neighborhood Neighborhood • Improvement: Angle criterion (Linsen) • Local Delaunay triangulation (Floater) – Project points onto tangent plane – Sort neighbors according to angle – Include more points if angle between subsequent points is above some threshold Point-Based Computer Graphics Surface Simplification – Project points into tangent plane – Compute local Voronoi diagram Mark Pauly Point-Based Computer Graphics Surface Simplification Covariance Analysis Covariance Analysis • Covariance matrix of local neighborhood N: • Consider the eigenproblem: C ⋅ v l = λl ⋅ v l , T ⎡ p i 1 − p ⎤ ⎡ p i1 − p ⎤ ⎥ ⎥ ⎢ ⎢ C = ⎢ L ⎥ ⋅ ⎢ L ⎥, ⎢p i − p ⎥ ⎢p i − p ⎥ ⎦ ⎦ ⎣ n ⎣ n • with centroid Point-Based Computer Graphics 1 p= N ∑p i ∈N l ∈ {0,1,2} ij ∈N • C is a 3x3, positive semi-definite matrix Ö All eigenvalues are real-valued Ö The eigenvector with smallest eigenvalue defines the leastsquares plane through the points in the neighborhood, i.e. approximates the surface normal i Surface Simplification Mark Pauly Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 2 Covariance Analysis Covariance Analysis • Covariance ellipsoid spanned by the eigenvectors scaled with corresponding eigenvalue • The total variation is given as: ∑p i ∈N − p = λ0 + λ1 + λ2 2 i • We define surface variation as: σ n (p ) = λ0 , λ0 + λ1 + λ2 λ0 ≤ λ1 ≤ λ2 – Measures the fraction of variation along the surface normal, i.e. quantifies how strong the surface deviates from the tangent plane Ö estimate for curvature Point-Based Computer Graphics Surface Simplification Mark Pauly Covariance Analysis Point-Based Computer Graphics Surface Simplification Mark Pauly Surface Simplification • Comparison with curvature: • Hierarchical clustering • Iterative simplification • Particle simulation original mean curvature Point-Based Computer Graphics variation n=20 variation n=50 Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Hierarchical Clustering Hierarchical Clustering • Top-down approach using binary space partition: • 2D example • Split the point cloud if: Mark Pauly root – Size is larger than user-specified maximum or – Surface variation is above maximum threshold • Split plane defined by centroid and axis of greatest variation (= eigenvector of covariance matrix with largest associated eigenvector) covariance ellipsoid split plane centroid • Leaf nodes of the tree correspond to clusters • Replace clusters by centroid Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 3 Hierarchical Clustering Hierarchical Clustering • 2D example • 2D example Point-Based Computer Graphics Surface Simplification Mark Pauly Hierarchical Clustering Point-Based Computer Graphics Surface Simplification Mark Pauly Hierarchical Clustering • 2D example 43 Clusters Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics 436 Clusters 4,280 Clusters Surface Simplification Hierarchical Clustering Iterative Simplification • Adaptive Clustering • Iteratively contracts point pairs Mark Pauly Ö Each contraction reduces the number of points by one • Contractions are arranged in priority queue according to quadric error metric (Garland and Heckbert) • Quadric measures cost of contraction and determines optimal position for contracted sample • Equivalent to QSlim except for definition of approximating planes Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 4 Iterative Simplification Iterative Simplification • Quadric measures the squared distance to a set of planes defined over edges of neighborhood – plane spanned by vectors e1 = p i − p and e 2 = e1 × n • 2D example • Compute initial point-pair contraction candidates • Compute fundamental quadrics n e2 p • Compute edge costs e1 pi Point-Based Computer Graphics Surface Simplification Mark Pauly Iterative Simplification • 2D example Point-Based Computer Graphics • 2D example priority queue cost edge cost 6 0.02 6 0.02 2 0.03 2 0.03 14 0.04 14 0.04 5 0.04 5 0.04 9 0.09 9 0.09 1 0.11 1 0.11 13 0.13 13 0.13 3 0.22 3 0.22 11 0.27 11 0.27 10 0.36 10 0.36 7 0.44 7 0.44 4 0.56 4 0.56 Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly Iterative Simplification • 2D example priority queue Surface Simplification Mark Pauly edge Iterative Simplification Point-Based Computer Graphics Surface Simplification Iterative Simplification priority queue • 2D example Point-Based Computer Graphics priority queue edge cost edge cost 6 0.02 2 0.03 2 0.03 14 0.04 14 0.04 5 0.06 5 0.06 9 0.09 9 0.09 1 0.11 1 0.11 13 0.13 13 0.13 3 0.23 3 0.23 11 0.27 11 0.27 10 0.36 10 0.36 7 0.49 7 0.49 4 0.56 4 0.56 Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 5 Iterative Simplification • 2D example Point-Based Computer Graphics Iterative Simplification • 2D example priority queue cost edge cost 2 0.03 14 0.04 14 0.04 5 0.06 5 0.06 9 0.09 9 0.09 1 0.11 1 0.11 13 0.13 13 0.13 3 0.23 3 0.23 11 0.27 11 0.27 10 0.36 10 0.36 7 0.49 7 0.49 4 0.56 4 0.56 Surface Simplification Mark Pauly Iterative Simplification • 2D example Point-Based Computer Graphics Surface Simplification priority queue edge cost 11 0.27 10 0.36 7 0.49 4 0.56 Surface Simplification Mark Pauly Point-Based Computer Graphics simplified model (2,000 points) Particle Simulation • Resample surface by distributing particles on the surface • Initialization • Particles move on surface according to interparticle repelling forces • Repulsion • Particle relaxation terminates when equilibrium is reached (requires damping) Mark Pauly – Randomly spread particles – Linear repulsion force Fi (p ) = k (r − p − p i ) ⋅ (p − p i ) Öonly need to consider neighborhood of radius r • Projection – Keep particles on surface by projecting onto tangent plane of closest point – Apply full MLS projection at end of simulation • Can also be used for up-sampling! Surface Simplification remaining point pair contraction candidates Surface Simplification Particle Simulation Point-Based Computer Graphics Mark Pauly Iterative Simplification original model (296,850 points) Point-Based Computer Graphics priority queue edge Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 6 Particle Simulation Particle Simulation • 2D example • 2D example • Initialization – randomly spread particles Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Particle Simulation Particle Simulation • 2D example • 2D example • Initialization – randomly spread particles • Initialization – randomly spread particles • Repulsion – linear repulsion force • Repulsion – linear repulsion force Fi (p ) = k (r − p − p i ) ⋅ (p − p i ) Point-Based Computer Graphics Surface Simplification Mark Pauly Fi (p ) = k (r − p − p i ) ⋅ (p − p i ) Point-Based Computer Graphics Surface Simplification Particle Simulation Particle Simulation • 2D example • 2D example Mark Pauly • Initialization – randomly spread particles • Initialization – randomly spread particles • Repulsion – linear repulsion force • Repulsion – linear repulsion force Fi (p ) = k (r − p − p i ) ⋅ (p − p i ) Fi (p ) = k (r − p − p i ) ⋅ (p − p i ) • Projection – project particles onto surface • Projection – project particles onto surface Point-Based Computer Graphics Mark Pauly Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 7 Particle Simulation Particle Simulation • Adaptive simulation • User-controlled simulation – Adjust repulsion radius according to surface variation samples in regions of high variation variation estimation Point-Based Computer Graphics Ö more simplified model (3,000 points) Surface Simplification – Adjust repulsion radius according to user input uniform Mark Pauly Particle Simulation Point-Based Computer Graphics original selective Surface Simplification Mark Pauly Measuring Error • Measure the distance between two point-sampled surfaces using a sampling approach • Maximum error: ∆ max (S, S ′) = max q∈Q d (q, S ′) Ö Two-sided Hausdorff distance • Mean error: ∆ avg (S, S ′) = 1 Q ∑ d (q,S′) q∈Q Ö Area-weighted integral of point-to-surface distances • Q is an up-sampled version of the point cloud that describes the surface S Point-Based Computer Graphics Surface Simplification Mark Pauly Measuring Error Point-Based Computer Graphics Surface Simplification Mark Pauly Measuring Error • d(q,S) approximates the distance of point q to surface S using the MLS projection operator original Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics simplified upsampled Surface Simplification error Mark Pauly 8 Comparison Comparison • Error estimate for Michelangelo’s David simplified from 2,000,000 points to 5,000 points • Execution time as a function of input model size (reduction to 1%) time (sec) hierarchical clustering 500 iterative simplification 450 400 particle simulation 350 300 250 200 150 100 50 0 0 500 1000 1500 2000 2500 3000 3500 input size Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Comparison Comparison • Execution time as a function of target model size (input: dragon, 535,545 points) • Summary time (sec) 70 hierarchical clustering 60 iterative simplification 50 particle simulation 40 30 20 Efficiency Mark Pauly Surface Control Error Implementation Hierarchical Clustering + - - + Iterative Simplification - + o o Particle Simulation o + + - 10 0 180 160 140 120 100 80 60 40 20 0 target size Point-Based Computer Graphics Surface Simplification Mark Pauly Point-based vs. Mesh Simplification point-based simplification with subsequent mesh reconstruction Point-Based Computer Graphics Surface Simplification Mark Pauly References mesh reconstruction with subsequent mesh simplification (QSlim) • Pauly, Gross: Efficient Simplification of Point-sampled Surfaces, IEEE Visualization 2002 • Shaffer, Garland: Efficient Adaptive Simplification of Massive Meshes, IEEE Visualization 2001 • Garland, Heckbert: Surface Simplification using Quadric Error Metrics, SIGGRAPH 1997 • Turk: Re-Tiling Polygonal Surfaces, SIGGRAPH 1992 • Alexa et al. Point Set Surfaces, IEEE Visualization 2001 Ö point-based simplification saves an expensive surface reconstruction on the dense point cloud! Point-Based Computer Graphics Surface Simplification Mark Pauly Point-Based Computer Graphics Surface Simplification Mark Pauly 9 Overview • Introduction • Pointshop3D System Components – Point Cloud Parameterization – Resampling Scheme – Editing Operators An Interactive System for Point-based Surface Editing (Part I) • Summary • See course notes Markus Gross Pointshop 3D Concept • Interactive system for point-based surface editing • Generalize 2D photo editing concepts and functionality to 3D point-sampled surfaces • Use 3D surface pixels (surfels) as versatile display and modeling primitive Input Surface editing Output irregular pointsampled model Pointshop 3D irregular pointsampled model Does not require intermediate triangulation Key Components • Point cloud parameterization Φ Ψ – creates one-to-one correspondence of surface and brush samples • Editing operator Ω u Resampling Editing Operator • Constrained minimum distortion parameterization of point clouds ⎡ x(u) ⎤ u ∈ [0,1] ⇒ X (u) = ⎢⎢ y (u)⎥⎥ = x ∈ P ⊂ R 3 ⎢⎣ z (u) ⎥⎦ 2 – combines surface and brush samples S ′ = Ω(Ψ (Φ ( S )), Ψ ( B )) modified surface v Parameterization – brings surface and brush into common reference frame • Dynamic resampling Parameterization original surface brush 1 Parameterization Parameterization • Find mapping X that minimizes objective function: brush points C( X ) = X (p ) − x ) ∑ ({ 2 j j∈M contraints = matching of feature points minimum distortion = maximum smoothness Parameterization surface points j + ε ∫ γ (u)du P { fitting constraints distortion Parameterization • Measuring distortion • Discrete formulation: θ 2 ⎛ ∂2 ⎞ X u (θ , r ) ⎟⎟ dθ 2 ∂ r ⎠ θ ⎝ γ (u) = ∫ ⎜⎜ r u – Integrates squared curvature using local polar reparameterization ~ C (U ) = ∑ (p j∈M j n ⎛ ∂U (x i ) ∂U (x i ) ⎞ − u j )2 + ε ∑ ∑ ⎜ − ~ ⎟ ⎜ ∂v ∂v j ⎟⎠ i =1 j∈N i ⎝ j 2 – Approximation: mapping is piecewise linear ⎛ ⎡cos(θ )⎤ ⎞ X u (θ , r ) = X ⎜⎜ u + r ⎢ ⎥ ⎟⎟ ⎣ sin(θ ) ⎦ ⎠ ⎝ Parameterization Parameterization • Directional derivatives as extension of divided differences based on k-nearest neighbors • HIerarchical solver for efficient computation of resulting sparse linear least squares problem 2 n ~ ⎛ ⎞ C (U ) = ∑ ⎜ b j − ∑ a j ,i u i ⎟ = b − Au j ⎝ i =1 ⎠ 2 2 Reconstruction Reconstruction • Parameterized scattered data approximation fitting functions ∑ Φ (u)r (u) X (u) = ∑ r (u) i weight functions i i i normalization i • Fitting functions – Compute local fitting functions using local parameterizations – Map to global parameterization using global parameter coordinates of neighboring points reconstruction with linear fitting functions Reconstruction Sampling • Reconstruction with linear fitting functions is equivalent to surface splatting! • Different sampling strategies: – Use the surface splatting renderer to reconstruct our surface function (see chapter on rendering) • Provides: • Fast evaluation • Anti-aliasing (Band-limit the weight functions before sampling using Gaussian low-pass filter) – Distortions of splats due to parameterization can be computed efficiently using local affine mappings – Resample the brush, i.e., sample at the original surface points – Resample the surface, i.e., sample at the brush points – Adaptive resampling, i.e., sample at surface or brush points depending on the respective sampling density • Dynamic sampling: see following chapter (Mark Pauly) Editing Operators Editing Operators • Painting • Sculpting – Texture, material properties, transparency weight functions in parameter space – Carving, normal displacement texture map displacement maps carved and texture mapped point-sampled surface 3 Editing Operators • Engraving surface detail Editing Operators Editing Operators • Filtering appearance and geometry Advanced Processing • Multiscale feature extraction • Filtering appearance and geometry – Scalar attributes, geometry Summary Reference • Pointshop3D provides a versatile platform for research in point based graphics • Uses points (3-dimensional pixels) as a graphics primitive • Generalizes 2D image editing tools to point sampled geometry • A variety of plug-ins for model cleaning, filtering, watermarking etc. • Version 2 supports advanced modeling operations. More to come… • Zwicker, Pauly, Knoll, Gross: Pointshop3D: An interactive system for Point-based Surface Editing, SIGGRAPH 2002 • check http://graphics.ethz.ch/pointshop3d/ 4 Motivation • 3D content creation pipeline Shape Modeling Mark Pauly Stanford University Point-Based Computer Graphics Motivation • Surface representations – Explicit surfaces (B-reps) • Polygonal meshes • Subdivision surfaces • NURBS – Explicit surfaces (B-reps) ☺ - Efficient rendering ☺ - Sharp features ☺ - Intuitive editing • Implicit surfaces • Polygonal meshes • Subdivision surfaces • NURBS - Boolean operations - Changes of topology - Extreme deformations • Implicit surfaces • Level sets • Radial basis functions • Algebraic surfaces Shape Modeling ☺ - Boolean operations ☺ - Changes of topology ☺ - Extreme deformations Mark Pauly Motivation • Level sets • Radial basis functions • Algebraic surfaces Point-Based Computer Graphics Shape Modeling - Efficient rendering - Sharp features - Intuitive editing Mark Pauly Motivation • Surface representations • Point cloud representation – Explicit surfaces (B-reps) • Polygonal meshes • Subdivision surfaces • NURBS • Implicit surfaces • Level sets • Radial basis functions • Algebraic surfaces Point-Based Computer Graphics Mark Pauly Motivation • Surface representations Point-Based Computer Graphics Shape Modeling Shape Modeling • Hybrid Representation • Explicit cloud of point samples • Implicit dynamic surface model Mark Pauly – Minimal consistency requirements for extreme deformations (dynamic re-sampling) – Fast inside/outside classification for boolean operations and collision detection – Explicit modeling and rendering of sharp feature curves – Integrated, intuitive editing of shape and appearance Point-Based Computer Graphics Shape Modeling Mark Pauly 1 Interactive Modeling Surface Model • Interactive design and editing of pointsampled models • Goal: Define continuous surface from a set of discrete point samples – Shape Modeling • Boolean operations • Free-form deformation – Appearance Modeling • Painting & texturing • Embossing & engraving Point-Based Computer Graphics continuous surface S interpolating or approximating P discrete set of point samples P = { pi, ci, mi, ... } Shape Modeling Mark Pauly Surface Model Point-Based Computer Graphics Shape Modeling Mark Pauly Boolean Operations • Moving least squares (MLS) approximation (Levin, Alexa et al.) + – Surface defined as stationary set of projection operator ΨP implicit surface model { SP = x ∈ R 3 ΨP ( x ) = x - - } – Weighted least squares optimization • Gaussian kernel function – local, smooth – mesh-less, adaptive Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations Boolean Operations • Create new shapes by combining existing models using union, intersection, or difference operations • Powerful and flexible editing paradigm mostly used in industrial design applications (CAD/CAM) • Easily performed on implicit representations Mark Pauly – Requires simple computations on the distance function • Difficult for parametric surfaces – Requires surface-surface intersection • Topological complexity of resulting surface depends on geometric complexity of input models Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly 2 Boolean Operations Boolean Operations • Point-Sampled Geometry • Classification: – Classification – given a smooth, closed surface S and point p. Is p inside or outside of the volume V bounded by S? • Inside-outside test using signed distance function induced by MLS projection – Sampling S V p • Compute exact intersection of two MLS surfaces to sample the intersection curve – Rendering • Accurate depiction of sharp corners and creases using point-based rendering Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations Boolean Operations • Classification: • Classification: – given a smooth, closed surface S and point p. Is p inside or outside of the volume V bounded by S? p 1. find closest point q on S – given a smooth, closed surface S and point p. Is p inside or outside of the volume V bounded by S? p 1. find closest point q on S 2. d=(p-q)·n defines signed distance of p to S S V q Point-Based Computer Graphics Shape Modeling Mark Pauly Boolean Operations Point-Based Computer Graphics S V n q Shape Modeling Mark Pauly Boolean Operations • Classification: • Classification: – given a smooth, closed surface S and point p. Is p inside or outside of the volume V bounded by S? 1. find closest point q on S 2. d=(p-q)·n defines signed distance of p to S 3. classify p as – inside V, if d < 0 – outside V, if d > 0 Point-Based Computer Graphics Mark Pauly Shape Modeling – represent smooth surface S by point cloud P S p V n P S V q Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly 3 Boolean Operations Boolean Operations • Classification: • Classification: – represent smooth surface S by point cloud P 1. find closest point q in P p – represent smooth surface S by point cloud P S P 1. find closest point q in P 2. classify p as – inside V, if (p-q)·n < 0 – outside V, if (p-q)·n > 0 V n q Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations Boolean Operations • Classification: • Classification: – piecewise constant surface approximation leads to false classification close to the surface Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations • Classification: • Classification: Point-Based Computer Graphics Shape Modeling Mark Pauly p V n q Mark Pauly – piecewise constant surface approximation leads to false classification close to the surface Boolean Operations – piecewise constant surface approximation leads to false classification close to the surface S P Mark Pauly – piecewise constant surface approximation leads to false classification close to the surface Point-Based Computer Graphics Shape Modeling Mark Pauly 4 Boolean Operations Boolean Operations • Classification: • Classification: – piecewise constant surface approximation leads to false classification close to the surface – use MLS projection of p for correct classification n ΨP (p ) p Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations Boolean Operations • Sampling the intersection curve • Newton scheme: Mark Pauly 1. identify pairs of closest points Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations Boolean Operations • Newton scheme: • 1. identify pairs of closest points Mark Pauly Newton scheme: 1. identify pairs of closest points 2. compute closest point on intersection of tangent spaces r q1 Point-Based Computer Graphics q1 q2 Shape Modeling Mark Pauly Point-Based Computer Graphics q2 Shape Modeling Mark Pauly 5 Boolean Operations Boolean Operations • • Newton scheme: Newton scheme: 1. identify pairs of closest points 2. compute closest point on intersection of tangent spaces 3. re-project point on both surfaces 1. identify pairs of closest points 2. compute closest point on intersection of tangent spaces 3. re-project point on both surfaces 4. iterate q′1 q′1 q′2 r′ q′2 r q1 Point-Based Computer Graphics q2 Shape Modeling Mark Pauly Boolean Operations Point-Based Computer Graphics Shape Modeling Mark Pauly Boolean Operations • Rendering sharp creases • Rendering sharp creases – represent points on intersection curve with two surfels that mutually clip each other Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations Boolean Operations • Rendering sharp creases • Rendering sharp creases Mark Pauly – easily extended to handle corners by allowing multiple clipping Difference Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Union Shape Modeling Mark Pauly 6 Boolean Operations Boolean Operations • Rendering sharp creases • Rendering sharp creases Difference Point-Based Computer Graphics Difference Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly Boolean Operations Boolean Operations • Rendering sharp creases • Boolean operations can create intricate shapes with complex topology A+B A⋅B A−B B−A Difference Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Boolean Operations Particle-based Blending • Singularities lead to numerical instabilities (intersection of almost parallel planes) • Boolean operations create sharp intersection curves • Particle simulation to create smooth transition Mark Pauly – Repelling force to control particle distribution – Normal potentials to control particle orientation Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly 7 Free-form Deformation Free-form Deformation • Smooth deformation field F:R3→R3 that warps 3D space • Can be applied directly to point samples Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly Free-form Deformation Free-form Deformation • How to define the deformation field? • Intuitive editing paradigm using painting metaphor Painting metaphor – Define rigid surface part (zero-region) and handle (one-region) using interactive painting tool – Displace handle using combination of translation and rotation – Create smooth blend towards zero-region • How to detect and handle selfintersections? Point-based collision detection, boolean union, particle-based blending • How the handle strong distortions? Dynamic re-sampling Point-Based Computer Graphics Shape Modeling Mark Pauly Free-form Deformation Point-Based Computer Graphics Shape Modeling Mark Pauly Free-form Deformation • Definition of deformation field: – Continuous scale parameter tx zero-region • tx = β (d0 / (d0 + d1)) • d0 : distance of x to zero-region • d1 : distance of x to one-region one-region d0 x d1 – Blending function • β : [0,1] → [0,1] • β ∈C0, β (0) = 0, β (1) = 1 original surface Point-Based Computer Graphics Shape Modeling – tx = 0 if x in zero-region – tx = 1 if x in one-region deformed surface Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly 8 Free-form Deformation Free-form Deformation • Definition of deformation field: • Translation for three different blending functions – Deformation function • F (x) = FT (x) + FR (x) d0 x d1 – Translation • FT (x) = x + tx · v – Rotation • FR (x) = M(tx) · x Point-Based Computer Graphics deformed surface blending function Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly Free-form Deformation Free-form Deformation • Rotational deformation along two different rotation axes • Embossing effect original surface Point-Based Computer Graphics color-coded scale parameter Shape Modeling bitmap image deformed surface zero- and one-regions deformed surface Mark Pauly Point-Based Computer Graphics Shape Modeling Collision Detection Collision Detection • Deformations can lead to self-intersections • Apply boolean inside/outside classification to detect collisions • Restricted to collisions between deformable region and zero-region to ensure efficient computations • Exploiting temporal coherence Point-Based Computer Graphics Point-Based Computer Graphics Shape Modeling Mark Pauly Shape Modeling Mark Pauly Mark Pauly 9 Collision Detection Dynamic Sampling • Interactive modeling session boolean union performed particle-based blending collision detected 10,000 points Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics 271,743 points Shape Modeling Dynamic Sampling Dynamic Sampling • Large model deformations can lead to strong surface distortions • Requires adaptation of the sampling density • Dynamic insertion and deletion of point samples • Surface distortion varies locally color-coded surface stretch Point-Based Computer Graphics Shape Modeling Mark Pauly Dynamic Sampling Shape Modeling surface after dynamic re-sampling Shape Modeling Mark Pauly Dynamic Sampling 1. Measure local surface stretch from first fundamental form 2. Split samples that exceed stretch threshold 3. Regularize distribution by relaxation 4. Interpolate scalar attributes Point-Based Computer Graphics Point-Based Computer Graphics Mark Pauly Mark Pauly • 2D illustration Point-Based Computer Graphics Shape Modeling Mark Pauly 10 Dynamic Sampling Free-form Deformation • Interactive modeling session with dynamic sampling original surface with zero- and one-regions final surface intermediate steps of deformation Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Results Results • 3D shape modeling functionality has been integrated into Pointshop3D to create a complete system for point-based shape and appearance modeling • Ab-initio design of an Octopus – – – – – – Mark Pauly – Free-form deformation with dynamic sampling from 69,706 to 295,222 points Boolean operations Free-form deformation Painting & texturing Sculpting Filtering Etc. Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Results Results • Modeling with synthetic and scanned data • Boolean operations on scanned data – Combination of free-form deformation with collision detection, boolean operations, particle-based blending, embossing and texturing Point-Based Computer Graphics Shape Modeling Mark Pauly Mark Pauly – Irregular sampling pattern, low resolution models Point-Based Computer Graphics Shape Modeling Mark Pauly 11 Results Conclusion • Interactive modeling with scanned data • Points are a versatile shape modeling primitive – noise removal, free-form deformation, cut-and-paste editing, interactive texture mapping Point-Based Computer Graphics Shape Modeling Mark Pauly – Combines advantages of implicit and parametric surfaces – Integrates boolean operations and free-form deformation – Dynamic restructuring – Time and space efficient implementations Point-Based Computer Graphics Shape Modeling Mark Pauly Conclusion References • Complete and versatile point-based 3D shape and appearance modeling system • Pauly: Point Primitives for Interactive Modeling and Processing of 3D Geometry, PhD Thesis, ETH Zurich, 2003 • Pauly, Keiser, Kobbelt, Gross: Shape Modeling with Point-sampled Geometry, SIGGRAPH 03 • Pauly, Kobbelt, Gross: Multiresolution Modeling with Point-sampled Geometry, ETH Technical Report, 2002 • Zwicker, Pauly, Knoll, Gross: Pointshop3D: An Interactive System for Point-based Surface Editing, SIGGRAPH 02 • Adams, Dutre: Boolean Operations on Surfel-Bounded Solids, SIGGRAPH 03 • Szeliski, Tonnesen: Surface Modeling with Oriented Particle Systems, SIGGRAPH 92 • www.pointshop3d.com – Directly applicable to scanned data – Suitable for low-cost 3D content creation and rapid proto-typing Point-Based Computer Graphics Shape Modeling Mark Pauly Point-Based Computer Graphics Shape Modeling Mark Pauly 12 S E C T I O N 4 SUPPLEMENTAL MATERIAL 4 Surface Representation with Point Samples. Marc Alexa Shape Modeling with Points. Mark Pauly Supplemental Notes on Surface Representation with Point Samples Marc Alexa Discrete Geometric Modeling Group Department of Computer Science Darmstadt U of Technology Fraunhoferstr. 5, 64283 Darmstadt, Germany phone: +49.6151.155.674, fax: +49.6151.155.669 Email: [email protected] April 26, 2004 This document contains supplemental notes on “Surface Representation with Point Samples” as part of the SIGGRAPH course on “Point-based Computer Graphics”. It should be used together with the slide set, which contains helpful illustrations. 1 Introduction Point sets are emerging as a surface representation. The particular appeal of points sets is their generality: every shape can be represented by a set of points on its boundary, where the degree of accuracy typically depends only on the number of points. Point sets do not have a fixed continuity class or are limited to a certain topology as many other surface representations. To define a manifold from the set of points, the inherent spatial interrelation among the points has to be exploited as implicit connectivity information. A mathematical definition or algorithm attaches a topology and a geometric shape to the set of points. This is non-trivial since it is unclear what spacing of points represents connected respectively disconnected pieces of the surface. Representing the surface with points is slightly different from the problem of reconstructing a surface from point samples: The basic idea of representation is to use the points as the main source of information about the shape. Efficient algorithms are applied to the points to determine if a certain point in space is inside or outside of the shape, how far it is from the surface, to 1 project this point onto the surface, or to intersect other primitives with the surface. In contrast, reconstruction is typically concerned with converting the point set into another representation, where these algorithmic goals are easier to perform. A consequence of this view is that we are interested in local algorithms. Only local algorithms have the premise to be efficient when used to perform certain local operations on very large point sets. Specifically, we’d like to avoid the construction of a global connectivity structure among the points. We admit that doing this can lead to very good reconstruction results, however, it also has some drawbacks, which we won’t discuss here. Despite the lack of global structure, we wish that putting all the local computations together would result in a smooth and (where reasonable) manifold surface. 2 Notation & Terms We assume that the points P = {pi ∈ R3 }, i ∈ {1, . . . , N}, are sampled from an unknown surface S , and that they might contain some noise due to the imperfect sampling process. Some sampling processes additionally provide normal information in each point, which we assume to be represented as N = {ni ∈ R3 , kni k = 1}. We assume that data is irregular, i.e. that the points are not sampled from a regular lattice in space. Our goal is to define computational methods for the interrogation or manipulation of a point x ∈ R3 . These computational tools indirectly define a surface Sˆ from the points P (and possibly the normals N ). We understand the term locality as the extent of space or the number of points that are necessary to perform the computations for x. A global method will potentially require all points in P to perform the computations. The reconstructed surface is said to be interpolating if P ∈ Sˆ, otherwise it is approximating. We will almost exclusively look at the case of approximation. Approximating the points takes into account that the surface is assumed to be not too wiggly and that the points contain some noise. An approximation allows smoothing this noise and providing a reasonably behaved surface. Before we approach the general surface representation problem, we’ll recall some basic methods for the interpolation or approximation of functional data. For this, we assume that each point pi = (qi , fi ) is composed of a position qi in parameter space (R2 in our setting) and a value fi at this position. 3 Interpolation and approximation of functional data For now, our goal is to determine a function f that interpolates or approximates the given constraints pi = (qi , fi ), i.e. fˆ(qi ) ≈ fi . Defining such a function means to describe an algorithm that computes for every x ∈ Rd a function value fˆ(x). We start with a very simple approach: given x, find the closest location for which a function value is defined, i.e. min j kq j − xk. If the minimum is not unique, choose the one with smallest 2 index j. Then set fˆ(x) to f j . More formally, we define fˆ(x) = q j , 0 < j < i ⇒ kq j − xk < kqi = xk, i < j < N ⇒ kq j − xk ≤ kqi = xk (1) The result is a function that interpolates the points but is not continuous. The obvious idea to improve the continuity of fˆ is to combine the values of several close points. In general, our approach looks like this: fˆ(x) = ∑ wi (x) fi , (2) i where wi (x) are weight functions appropriate to combine the values of several points in a location x. Depending on how the set of ’close’ points is identified and how the weight functions are compute based on the set of close points, several methods with different properties can be derived. We will take a close look at the following ideas: Voronoi techniques Identify the regions for which the location of data point qi is closest and exploit the adjacency of these regions. RBF Attach a (radial) function to each data point that describes how it influences space. Shepard Collect points in a certain radius and weight them based on distance. MLS Collect points in a certain radius and weight them so that the resulting function would reproduce a given class of functions. We will first look at these approaches and then broaden the discussion by introducing (adaptive) regular spatial subdivisions based on the last approach. 3.1 Voronoi techniques We note that a certain set of locations x has the property kq j − xk ≤ kqi − xk, j 6= i, i.e. is closest to q j . This set of points is the Voronoi cell C j , i.e. formally C j = {x, kq j − xk ≤ kqi = xk, j 6= i} (3) Note that two Voronoi cells Ci and C j might have some points in common (points x for which kqi − xk = kq j − xk. If Ci ∩C j 6= 0/ we call Ci and C j or, equivalently, u and j neighbors. We can use the concept of Voronoi cells and neighboring data points in different ways to improve the continuity of fˆ. We will only sketch these ideas as it turns out that computing Voronoi cells is a global problem. Connecting all neighboring locations qi and q j defines a geometric graph, the so called Delaunay graph. In most cases all faces of this graph are triangles. Triangulating the remaining faces with 4 or more edges generates the Delaunay triangulation. Let {Ti, j,k } be the triangles spanned by qi , q j , qk . Note that the signed area of a triangle spanned by x1 , x2 , x3 is A(x1 , x2 , x3 ) = 1 det (x1 , x2 , x3 ) 2 3 (4) The weights for defining fˆ are barycentric coordinates for the triangle that contains x. To compute those, we need to identify (i, j, k) so that x ∈ Ti, j,k . To compute this, note that x ∈ Ti, j,k ⇐⇒ sgnA(x, qi , q j ) = sgnA(x, q j , qk ) = sgnA(x, qk , qi ). (5) Then, barycentric coordinates w.r.t. the locations qi , q j , qk are defined as w(x)i = A(x, q j , qk ) A(x, qi , q j ) A(x, qk , qi ) , w(x) j = , w(x)k = . A(qi , q j , qk ) A(qi , q j , qk ) A(qi , q j , qk ) (6) In essence, fˆ is defined as the Delaunay triangulation of the locations {qi } lifted to the heights { fi }. As such, fˆ is a continuous function, with discontinuous derivatives along the edges of the triangulation. The Voronoi cells can be further exploited to achieve an interpolating function fˆ with continuous derivatives. To derive the weights for x, compute the Voronoi cell Cx for x if it was added to the set of points. Then weights could be defined as wi (x) = |Cx ∩Ci | , |Cx | (7) i.e. as the relative area the cell Cx shares with the original cells {Ci }. There are alternative definitions for the weights based on this general concept, leading to additional properties for fˆ. 3.2 Radial Basis Functions A basic and very general approach is to model the weight functions as radial functions wi (x) = ci θ (kx − qi k) ⇔ wi (x) fi = ci θ (kx − qi k), fi (8) where θ is a function that describes the influence of qi on x based on the distance between the two locations. In this approach, all weight functions wi are essentially the same and only differ by a linear factor. Note that the method is already fully defined for a fixed function θ in case we ask for interpolation: Requiring fˆ(q j ) = f j leads to fˆ(q j ) = ∑ ci θ (kq j − qi k) = f j , (9) i which is, in fact, a system of linear equations: θ (kq0 − q0 k) θ (kq0 − q1 k) θ (kq0 − q2 k) θ (kq1 − q0 k) θ (kq1 − q1 k) θ (kq1 − q2 k) θ (kq2 − q0 k) θ (kq2 − q1 k) θ (kq2 − q2 k) .. .. .. . . . ··· c0 f0 · · · c1 f1 = · · · c2 f2 .. .. .. . . . (10) So, before we are able to compute the weights we first need to solve this linear system. This requires that the system has a solution, which means the data points allow being interpolated 4 with the given functions. As the matrix depends only on values of θ , solvability obviously depends on the choice of θ . Standard choices for the radial function θ are θ (δ ) = δ −ν or the Gaussian θ (δ ) = exp(δ 2 /h2 ). However, another concern makes these functions impractical: Each point influences every other point, making the approach global. This can also be recognized from the fact that the a dense linear system has to be solved before fˆ could be evaluated in any point. In an attempt to make the solution local we should use locally supported radial functions. This means, we can choose a distance parameter ε. If two points are further apart then ε the function θ attached to either of them vanishes in the other point, i.e. δ > ε ⇒ θ (δ ) = 0. Popular choices with good properties are Wendland’s radial functions [Wen95], because they consist of polynomial pieces with low degree (i.e. they are easy to compute) and lead to solvable linear systems. The particular function to be used depends on the space in which the locations qi live. Using these locally supported functions leads to sparse linear systems, which can be solved in almost linear time. Nevertheless, strictly speaking this is a global solution, as the inverse of a sparse matrix is not necessarily sparse. Practically speaking this means moving one point could potentially influence points further away then ε by a cascade of effects on other points. On the other hand, the sparse linear system has to be solved only once. This defines the linear factors {ci }, which in turn define the weight functions wi (x). Evaluating fˆ(x) is typically very cheap, as θ has to be evaluated only for few close points. 3.3 Least squares fitting I assume that most readers are familiar with the idea of least squares fitting a polynomial to given data. Here, we will rediscover this method in our setting, by introducing the concept of a precision set. I hope this presentation eases the understanding of the following techniques. As before, we represent fˆ at x as ∑i wi (x) fi . We ask that fˆ has a certain precision, which is described by a precision set G : If the pairs (qi , fi ) happen to be sampled from a function contained in the precision set (say g ∈ G ), then we wish that fˆ results to be exactly that function. We can formalize this requirement as follows: For every g ∈ G the weight functions have to satisfy g(x) = ∑ wi (x)g(qi ). (11) i As a more concrete example, consider the precision set of quadratic polynomials g(x) = a + bT x + xT Cx. Look at the following system of equations 1 = ∑ wi (x)1 i x0 = ∑ wi (x)qi0 i .. . (12) x02 = ∑ wi (x)q2i0 i .. . 5 and note that the set of linear combinations of these equations a + bT x + xT Cx = ∑ wi (x) a + bT qi + qT i Cqi . (13) i is, in fact, the requirement of reproducing any function from the precision set of quadratic polynomials. We can write the system of equations in matrix form as Qw(x) = z. (14) Typically, we will have more points than dimensions in the space of polynomials, i.e. the system is underdetermined. We need to restrict the weights further. A common way to do this would be to ask that sum of squared weights is minimal, i.e. min w(x)T w(x). ∑ wi (x) = min w(x) {wi (x)} i (15) How could we find this minimum, subject to the linear constraints given in Eq. 14? Assume we know the solution vector w(x). Now look at the polynomial (a, b0 , . . .)Qw(x). We can certainly choose the polynomial coefficients (a, b0 , . . .) so that this polynomial attains a minimum or a maximum for the given weight vector w(x). So instead of minimizing only squared weights, we try to minimize w(x)T w(x) − (a, b0 , . . .)Qw(x), (16) where we have the polynomial coefficients as additional degrees of freedom. This approach helps to include the linear constraints in the minimization, at the cost of additional variables to solve for. A necessary condition for the minimum is that all partial derivatives are identical zero. Taking all partial derivatives w.r.t. the weights and setting to zero leads to w(x)T − (a, b0 , . . .)Q = 0 ⇐⇒ w(x) = QT (a, b0 , . . .)T (17) Using that in Eq. 14 yields QQT (a, b0 , . . .)T = z, (18) which is identical to the normal equation for least squares fitting a polynomial and also shows that the solution is independent of the location x. Once the polynomial coefficients are determined one could indeed solve for the weights at x, however, in this case it is easier to compute fˆ using the representation as a polynomial. Notice that the solution we have presented works for any precision set that could be represented as finite linear space. 3.4 Moving Least Squares We will follow the basic ideas of the last section. The only modification is that we localize weights. We do this by incorporating a separation measure into the minimization of squared weights: min ∑ w2i (x)η(kqi − xk) = min w(x)T E(x)w(x) (19) {wi (x)} i w(x) 6 The separation measure η(kqi − xk) penalizes the influence of points at qi far away from x, i.e. the function increases with the distance between qi and x. The solution to this constrained minimization is similar to the uniform situation. Now one has to solve w(x)T E(x) − (a, b0 , . . .)Q = 0 (20) which leads to w(x) = E(x)−1 QT (a, b0 , . . .)T . (21) This can be inserted into the constraint equation Qw(x) = z to get the polynomial coefficients: QE(x)−1 QT (a, b0 , . . .)T = z (22) We see that the polynomial coefficients result from a weighted least squares system. The weighting comes from the η −1 , which we call θ for convenience. It depends on the location x, because η depends on x. The resulting approach could also be interpreted like this: In each location x determine a locally weighted least squares fitting polynomial and use the value of this polynomial at x (a ’moving’ least squares approximation [LS98, Lev98]). In this interpretation it seems the approximating values of fˆ(x) are computed independently for different locations, so it might not be immediately clear that fˆ is a continuously differentiable function. Our derivation of fˆ, however, reveals that it is, if η (or, better, E(x)) is continuously differentiable. If θ is locally supported (i.e. vanishes for large distances between x and qi ) the computations for x are also local, as they depend only on the data points that are within the support. For η(0) = 0 (i.e. θ (0) = ∞) the resulting function fˆ interpolates the points. Notice that the statements about the continuity of fˆ hold also for the case of local support and/or interpolation. The coefficients of the polynomial could be used to find the weights as −1 w(x) = E(x)−1 Q QE(x)−1 QT z. (23) Now we take a closer look at the special case of asking only for constant precision. Then, Q is a row vector containing only ones and z = 1. Then E(x)−1 Q is a row vector containing the terms θ (kqi − xk) and QE(x)−1 QT is the sum of these terms. This means we get the following weights for location x when asking only for constant precision: w j (x) = θ (kq j − xk) ∑i θ (kqi − xk) (24) This type of weight is commonly called a Partition of Unity, because the weights sum up to one everywhere. Using θ (δ ) = δ −r , r > 0 we rediscover a particular and well-known instance of Partition of Unity: Shepard’s interpolation method [She68, FN80]. 3.5 Adaptive spatial subdivisions The MLS method associates a local approximation with each location in the parameter domain. For the evaluation of fˆ(x) this means one has to compute the polynomial coefficients (or, the weights wi ) for each location x. An approach to reduce the amount of computations is to divide 7 the parameter space into cells and use only one approximating function per cell. Of course, to achieve an overall continuous approximation we make the cells slightly overlapping and blend the local approximations in the areas of overlap. For this blending, we use a Partition of Unity, however, now derived from the cells rather than the points. The benefit of computing the local approximations and the weights for the cells is simply that we can construct the cells so that we have much less cells than points. Because of blend function that sum up to one are a natural choice this approach is called Partition of Unity approach in some other communities [BM97, GS00, GS02a, GS02b]. To be more concrete, let the cells be denoted Ωi . Together they form an overlapping covering of the parameter domain, i.e. the intersection of neighboring cells has non-zero area. With each cell Ωi we associate a weight function wi and a local (polynomial) approximation fˆi of the points within Ωi . Then the global approximating function is given as fˆ(x) = ∑ wi (x) fˆi (25) i How do we compute the local approximations fˆi and the weights wi (x)? The local approximations could be simply low degree polynomials derived from least squares fitting to the points contained in Ωi . The weights could be computed by starting from a blend function θi that has support only inside the cell Ωi , i.e. vanishes outside the cell and then deriving w j (x) = θ (x)/ ∑i θi (x), very similar to the MLS situation. A good choice of θ depends on the shape of the cell. For spherical cells with radius Ri centered at ci the quadratic B-spline b(t) yields a reasonable low degree approximation of a Gaussian as θi (x) = b(kci − xk/Ri ). For box-shaped cells, the quadratic B-spline could be used in a tensor product. For example, given a box with center ci and edge lengths e0 , e1 , . . . this yields θi (x) = b(2kci0 − x0 k/e0 ) · b(2kci1 − x0 k/e1 ) . . .. Of course, other choices are possible. Having these ingredients together, it remains to find a reasonable decomposition of the parameter space into cells. here, reasonable means that each cell contains not too many points, so that the local approximation is accurate, but also not too few, because than the number of cells is large (and, the local approximation could become ambiguous). For irregular data this could be very difficult. A much better strategy is to use adaptive subdivisions: We start by subdividing the parameter domain into few cells. For example, we divide the space into half along each dimension and then slightly increase the size of the cells in all dimensions to generate the necessary overlap. For example, in the plane this generates a 2 by 2 grid of rectangular cells. In each of the cells we compute a local (low degree polynomial) approximation. Then we compute the max-norm error of these local approximations. If this error exceeds a certain threshold, this cell is subdivided again, and the procedure is applied to the sub-cells. In the plane, the result is a quadtree (though with slightly larger cells to provide the overlap), where the error of the local approximation is bounded by a given threshold in each cell. Once this is achieved, weight functions are determined for the cells, and fˆ could be evaluated. In fact, we don’t need to build all approximations before starting to evaluate fˆ – this can be done on the fly only where needed: For the evaluation of fˆ(x) we determine all cells containing x and compute local approximations. Then the error is determined and the cells are subdivided if necessary. This procedure is repeated for the sub-cells and so on, until the error is bounded. 8 For the cells with bounded error that contain x, fˆi (x) and wi (x) and determined to assemble fˆ. A pseudo-code for the evaluation of fˆi is given below. For evaluating fˆ this function has to be called with a cell that contains all points in P: Evaluate fˆ(Ωi , x, ε) if ( fˆi is not created yet) then Create fˆi and compute e = maxq j ∈Ωi k f j − fˆi (qi )k; if (e > ε) then Create subcells {ωk }; for each ωk Evaluate fˆ(ωk , x, ε); end for else (error is below bound) fˆ(x) = fˆ(x) + wi (x) ∗ fˆi (x); σw = σw + wi (x); end if fˆ(x) = fˆ(x)/σw ; This adaptive approach has several nice properties: The approximation error is bounded, because the max-norm error of fˆ cannot be larger than the max-norm error in any of the cells. This is a result of the weight functions summing up to one. All computations are purely local, resulting a an almost linear run time behavior. In fact, for practical distributions of points the algorithm scales linearly with the number of points. Another consequence of data locality is that out-of-core or parallel implementations are straightforward. 4 Normals So far we have considered functional data. Now we turn to the problem of approximating a surface represented by points P in space. In this setting, we don’t have a suitable parameter domain to directly apply the techniques explained above. It turns out, that almost all approaches compensate for this lack of information by using or approximating tangent planes or normals on or close to the point set. Normals might be part of the data, or not. If normals are missing, we can try to estimate them as follows: Assume we want to compute the normal n in a location q in space. The points close to q describe the surface around q. A tangent in q should be as close as possible to these close points. Determining a tangent plane around q can be formulated as a least squares problem. We search a plane H(x) : nT q = nT pi , knk = 1 passing through q that minimizes the squares (nT (q − pi ))2 . However, we want to consider only few points pi close to q. We could do this by either using only the k-nearest neighbors of q, or by weighting close points with a locally supported weight function θ . Because the k-nearest neighbor approach could be simulated by using a hat function with appropriate radius for θ , we will only detail the locally weighted version. Then, n is defined 9 by the following minimization problem: 2 min ∑ nT pi − q) θ (kpi − pq k) (26) |nk=1 i This is a non-linear optimization problem, because of the quadratic constraint knk = 1. To arrive at a computable solution, we use the outer product matrices (pi − q)(pi − q)T and rewrite the functional to be minimized as ! m(n) = nT ∑(pi − q)(pi − q)T θ (kpi − qk) n, (27) i and inspect the eigenvalue/eigenvector decomposition of the sum of outer products ∑(pi − q)(pi − q)T = Ediag(λ0 , λ1 , . . .)ET . (28) i Using this decomposition we see that in transformed coordinates ET n the functional m(n) = nT Ediag(λ0 , λ1 , . . .)ET n has only pure quadratic terms, and each of these quadratic terms has an eigenvalue as coefficient. Let λ0 ≤ λ1 , . . ., then m(n) clearly attains its minimum among all unit-length vectors for mvE T n = (1, 0, 0, . . .)T . This means, n is the eigenvector corresponding to the smallest eigenvalue. Fitting a tangent plane will only yield a normal direction, not an orientation. We assume that the correct orientation can be derived from inside/outside information generated using scanning the object. 5 Implicit surfaces from points and offset points The basic idea of approaches based on implicit surfaces is to assume that all points on the surface have zero value, i.e. the surface is implicitly defined by S = {x| fˆ(x) = 0}. (29) In this setting, the point set delivers a set of constraints of the form fˆ(pi ) = 0. (30) Now, our approach is to apply the techniques for local function estimation presented in the preceding section. However, all of these methods would result in fˆ = 0, as this perfectly satisfies all constraints. We obviously need additional non-zero constraints. These additional constraints have to be generated based on the given point data. The normals can be used to generate additional point constraints for fˆ. A standard trick is this: Move a small step (say δ ) from pi in normal direction outwards from the surface. This point is pi + δ ni . Require fˆ to be δ at this point. The surface could be additionally supported by also moving to the inside and requiring that the value at pi − δ ni is −δ . A potential danger of this approach is that the pi + δ ni is not really δ away from the surface, because we the step is so large that we have moved towards some other part of the surface. A 10 good strategy to check and avoid this is to compute the closest point to pi + δ ni . If this is no pi , the step size δ has to be reduced until this holds true. If a spatial subdivision is used to organize the points this is another good way to add non-zero constraints. In each of the corners and centers of a cell the distance to the surface is approximated as the smallest distance to any point of the set. A sign for the distance can be computed from inside/outside information. For small distances, the distance to the closest point becomes less reliable. We propose to rather compute the distances to the k-nearest points (k = 3) and check that they all have the same sign. The result of either procedure is a set of additional constraints of the form fˆ(pN+i ) = di . (31) Together with the constraints fˆ(pi ) = 0 they can be used to approximate a function fˆ using any of the techniques for approximating functions as described in the last section. Several works discuss the details of using RBF for approximating the implicit function [Mur91, MYR∗ 01, CBC∗ 01, DST01, TO02, DTS02, OBS03], spatial subdivisions have been used together with Partition of Unity weights, either RBF approximations in a k-d tree [TRS04], or local polynomial and specific sharp edge functions in the cells of an octree [OBA∗ 03]. 6 Implict surface from points and tangent frames Rather than generating additional point constraints to make the standard function approximation techniques applicable we could also try to adapt them to the more general setting. An underlying idea of several approaches in this direction is to estimate local tangent frames and then use a standard technique in this local tangent frame. 6.1 Hoppe’s and related approaches A simple approach to generate an implicit function fˆ based on the normals is due to Hoppe [HDD∗ 92]: For a point in space x compute the closest point pi in P. Then set fˆ(x) to ni (pi − x), i.e. the signed distance to the tangent plane through pi . This yields a piecewise linear approximation of signed distances to the surface. The set of points in space associated to the same point pi form the Voronoi cell around pi . So, another viewpoint on this is that we use local linear approximations, however, for Voronoi cells we use a local frame based on the tangent plane. One can compute a smoother surface approximation by exploiting the Voronoi cells around the points and using Voronoi interpolation as explained in the preceding section. This has been exploited by Boissonnat et al. [BC00]. 6.2 MLS surfaces The MLS surface SP of P is defined implicitly by a projection operator. The basic idea for projecting a point r onto SP is based on two steps: First, a locally tangent reference domain H is computed. Then, a local bivariate polynomial is fitted over H to the point set. 11 However, to compensate for points with some distance to the surface, we don’t restrict the tangent plane to pass through the point r. Yet, we still want to weight the influence of points based on the distance to the origin of the tangent frame. This leads to a more complex minimization problem, however, yields the desired projection property. Specifically, the local reference domain H = {x|hn, xi − D = 0, knk = 1 is determined by minimizing N ∑ (hn, pi − r − tni)2 θ (kpi − r − tnk) (32) i=1 among all normal directions n and offsets t. Let qi be the projection of pi onto H, and fi the height of pi over H, i.e fi = n · (pi − q). The polynomial approximation g is computed by minimzing the weighted least squares error N ∑ (g(xi , yi ) − fi )2 θ (kpi − r − tnk) (33) i=1 The projection of r is given by MLS(r) = r + (t + g(0, 0))n (34) Formally, the surface SP is the set of points that project onto themselves. We can also define the surface in the standard notation using fˆ(x) = k(t + g(0, 0))n(x)k (35) The projection procedure itself has turned out to be a useful computational method for computing points on the surface. We point the reader to [Lee00, ABCO∗ 01, PGK02, PKG02, ABCO∗ 03, FCOAS03, Lev03, PKKG03] for details on the properties, extensions, and implementation of this approach. 6.3 Surfaces from normals and weighted averages Inspired by MLS surfaces, we can also define the surface implicitly based on normal directions and weighted averages. The implicit function f : R3 → R describes the distance of a point x to the weighted average a(x) projected along the normal direction n(x): fˆ(x) = n(x) · (a(x) − x) (36) where the weighted average of points at a location s in space is a(s) = ∑N−1 i=0 θ (ks − pi k)pi . N−1 θ (ks − pi k) ∑i=0 (37) If θ is locally supported one has to make sure to compute fˆ only in the support of the weights. Computing the weighted average and the local tangent frame also allows to define boundaries of the surface in a natural way: We inspect the relative location of the weighted average a(x) in the points. For points far away from the point set the distance ||x − a(x)|| increases, while we 12 expect this distance to be rather small for locations close to (or “inside”) the points. The main idea for defining a boundary is to require ||x − a(x)|| to be less than a user-specified threshold. For computing points on the surface, several efficient curve-surface intersection and projection operators can be implemented. These intersections can be computed independent of normal orientation. More details can be found in [AA03b, AA03a, AA04] Acknowledgements Numerous people have contributed to this work. I gratefully acknowledge the help of Nina Amenta, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, Markus Gross, Leonidas Guibas, David Levin, Jörg Peters, Ulrich Reif, Claudio Silva, Denis Zorin, and several anonymous reviewers. This work was supported by a grants from the European Union, the German Ministry of Science and Education (BMBF), the Israeli Ministry of Science, the German Israeli Foundation (GIF), and the Israeli Academy of Sciences (center of excellence). 13 References [AA03a] A DAMSON A., A LEXA M.: Approximating and intersecting surfaces from points. In Proceedings of the Eurographics/ACM SIGGRAPH symposium on Geometry processing (2003), Eurographics Association, pp. 230–239. [AA03b] A DAMSON A., A LEXA M.: Ray tracing point set surfaces. In Proceedings of Shape Modeling International (2003). [AA04] A DAMSON A., A LEXA M.: Approximating bounded, non-orientable surfaces from points. In Proceedings of Shape Modeling International 2004 (2004), IEEE Computer Society. accepted for publication. [ABCO∗ 01] A LEXA M., B EHR J., C OHEN -O R D., F LEISHMAN S., L EVIN D., S ILVA C. T.: Point set surfaces. In Proceedings of the conference on Visualization ’01 (2001). [ABCO∗ 03] A LEXA M., B EHR J., C OHEN -O R D., F LEISHMAN S., L EVIN D., S ILVA C. T.: Computing and rendering point set surfaces. IEEE Transactions on Computer Graphics and Visualization 9, 1 (2003), 3–15. [BC00] B OISSONNAT J.-D., C AZALS F.: Smooth shape reconstruction via natural neighbor interpolation of distance functions. In Proceedings of the 16th Annual Symposium on Computational Geometry (SCG-00) (N. Y., June 12–14 2000), ACM Press, pp. 223–232. [BM97] BABUšKA I., M ELENK J. M.: The partition of unity method. International Journal of Numerical Methods in Engineering 40 (1997), 727–758. [CBC∗ 01] C ARR J. C., B EATSON R. K., C HERRIE J. B., M ITCHELL T. J., F RIGHT W. R., M C C ALLUM B. C., E VANS T. R.: Reconstruction and representation of 3d objects with radial basis functions. In Proceedings of ACM SIGGRAPH 2001 (2001), Computer Graphics Proceedings, Annual Conference Series, pp. 67–76. [DST01] D INH H. Q., S LABAUGH G., T URK G.: Reconstructing surfaces using anisotropic basis functions. In International Conference on Computer Vision (ICCV) 2001 (Vancouver, Canada, July 2001), pp. 606–613. [DTS02] D INH H. Q., T URK G., S LABAUGH G.: Reconstructing surfaces by volumetric regularization. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 10 (October 2002), 1358–1371. [FCOAS03] F LEISHMAN S., C OHEN -O R D., A LEXA M., S ILVA C. T.: Progressive point set surfaces. ACM Transactions on Graphics (TOG) 22, 4 (2003), 997–1011. [FN80] F RANKE R., N IELSON G.: Smooth interpolation of large sets of scattered data. International Journal of Numerical Methods in Engineering 15 (1980), 1691–1704. 14 [GS00] G RIEBEL M., S CHWEITZER M. A.: A Particle-Partition of Unity Method for the solution of Elliptic, Parabolic and Hyperbolic PDE. SIAM J. Sci. Comp. 22, 3 (2000), 853–890. [GS02a] G RIEBEL M., S CHWEITZER M. A.: A Particle-Partition of Unity Method—Part II: Efficient Cover Construction and Reliable Integration. SIAM J. Sci. Comp. 23, 5 (2002), 1655–1682. [GS02b] G RIEBEL M., S CHWEITZER M. A.: A Particle-Partition of Unity Method—Part III: A Multilevel Solver. SIAM J. Sci. Comp. 24, 2 (2002), 377–409. [HDD∗ 92] H OPPE H., D E ROSE T., D UCHAMP T., M C D ONALD J., S TUETZLE W.: Surface reconstruction from unorganized points. Computer Graphics (Proceedings of SIGGRAPH 92) 26, 2 (July 1992), 71–78. [Lee00] L EE I.-K.: Curve reconstruction from unorganized points. Computer Aided Geometric Design 17, 2 (February 2000), 161–177. [Lev98] L EVIN D.: The approximation power of moving least-squares. Math. Comp. 67 (1998), 1517–1531. [Lev03] L EVIN D.: Mesh-independent surface interpolation. Geometric Modeling for Scientific Visualization (2003). [LS98] L ANCASTER P., S ALKAUSKAS K.: Surfaces generated by moving least squares methods. Mathematics of Computation 37 (1998), 141–158. [Mur91] M URAKI S.: Volumetric shape description of range data using “Blobby Model”. Computer Graphics 25, 4 (July 1991), 227–235. Proceedings of ACM SIGGRAPH 1991. [MYR∗ 01] M ORSE B. S., YOO T. S., R HEINGANS P., C HEN D. T., S UBRAMANIAN K. R.: Interpolating implicit surfaces from scattered surface data using compactly supported radial basis functions. In Shape Modeling International 2001 (Genova, Italy, May 2001), pp. 89–98. [OBA∗ 03] O HTAKE Y., B ELYAEV A., A LEXA M., T URK G., S EIDEL H.-P.: Multi-level partition of unity implicits. ACM Transactions on Computer Graphics (SIGGRAPH 2003 Proceedings) 22, 3 (2003), 463–470. [OBS03] O HTAKE Y., B ELYAEV A., S EIDEL H.-P.: A multi-scale approach to 3d scattered data interplation with compactly supported basis functions. In Proceedings of Shape Modeling International 2003 (May 2003). [PGK02] PAULY M., G ROSS M., KOBBELT L. P.: Efficient simplification of point-sampled surfaces. In Proceedings of the conference on Visualization ’02 (2002), pp. 163– 170. 15 [PKG02] PAULY M., KOBBELT L., G ROSS M.: Multiresolution Modeling of Point-Sampled Geometry. Tech. rep., ETH Zurich, Department of Computer Science, 2002. [PKKG03] PAULY M., K EISER R., KOBBELT L. P., G ROSS M.: Shape modeling with pointsampled geometry. ACM Transactions on Graphics (TOG) 22, 3 (2003), 641–650. [She68] S HEPARD D.: A two dimensional interpolation function for irregular spaced data. In Proceedings of 23rd ACM National Conference (1968), pp. 517–524. [TO02] T URK G., O’B RIEN J.: Modelling with implicit surfaces that interpolate. ACM Transactions on Graphics 21, 4 (October 2002), 855–873. [TRS04] T OBOR I., R EUTER P., S CHLICK C.: Multiresolution reconstruction of implicit surfaces with attributes from large unorganized point sets. In Proceedings of Shape Modeling International (SMI 2004) (2004). in press. [Wen95] W ENDLAND H.: Piecewise polynomial, positive definite and compactly supported radial basis functions of minimal degree. Advances in Computational Mathematics 4 (1995), 389–396. 16 Shape Modeling with Points Supplemental Course Notes SIGGRAPH 2004 Mark Pauly May 12, 2004 2 Chapter 1 Local Surface Analysis Mark Pauly This part of the course notes describes some of the fundamental techniques that are used to deﬁne more sophisticated processing algorithms for point-based surfaces. In particular, eﬃcient methods for estimating local surface properties, such as normal vector and curvature, are described. These computations are based on local neighborhoods of point samples, which are deﬁned as subsets of a given point cloud P that satisfy some local neighborhood relation. Also, the moving least squares (MLS) surface model is discussed, which deﬁnes a smooth manifold surface from the point cloud and allows the evaluation of the signed distance function. The concept of a 2-manifold surface embedded in 3-space is crucial for the algorithms described in these notes. The following deﬁnition of a manifold surface is taken from [dC76]: Deﬁnition: A subset S ⊂ IR3 is a 2-manifold surface, if for each x ∈ S, there exists a neighborhood V in IR3 and a map X : U → V ∩ S of an open set U ⊂ IR2 onto V ∩ S ⊂ IR3 such that • X is diﬀerentiable, i.e., if X(u, v) = (x(u, v), y(u, v), z(u, v)) for (u, v) ∈ U then the functions x, y, and z have continuous partial derivatives of all orders in U , • X is a homeomorphism, i.e., the inverse X −1 : V ∩ S → U exists and is continuous, • for each u ∈ U , the diﬀerential dXu is one-to-one. Note that the deﬁnition of a surface is based on local (possibly inﬁnitesimal) neighborhoods. Intrinsic properties of the surface, such as tangent plane or Gaussian curvature, are deﬁned with respect to such neighborhoods (see [dC76] for details). 1.1 Local Neighborhoods In the discrete setting, a local neighborhood can be deﬁned through spatial relations of the sample points. Given a point p ∈ P , a local neighborhood is deﬁned as an index set Np , such that each pi , i ∈ Np satisﬁes a certain neighborhood condition. This condition should be set in such a way that the points of Np adequately represent a small, local surface patch around the point p. For the computations described below it is important that local neighborhoods only depend on the geometric locations of the point samples in space, not on some global combinatorial structure associated with the point cloud. If the point cloud is locally uniform, the k-nearest neighbors relation is most commonly used to deﬁne local neighborhoods. Other relations, e.g., local Delaunay neighbors [FR01], or Linsen’s method [Lin01] can be used for non-uniformly sampled models. 3 CHAPTER 1. LOCAL SURFACE ANALYSIS 4 1.1.1 Local Sampling Density An important measure for analyzing point-sampled surfaces is the local density of the point samples, i.e., the number of samples per unit area. The local sampling density ρi at a sample point pi ∈ P can be estimated by computing the sphere with minimum radius ri centered at pi that contains the k-nearest neighbors to pi . By approximating the intersection of the underlying surface and this sphere with a disc, ρi can be deﬁned as ρi = k/(πri2 ). This discrete estimation of local sampling density can be extended to a continuous function ρ : IR3 → IR+ using scattered data approximation. A continuous density function ρ(x) is useful because it provides for each point x on the surface S an estimate of the sampling density of a small patch around x. Similarly, an estimate for the local sample spacing can be derived as η(x) = 1/ ρ(x). η measures the average distance of sample points within the sphere deﬁned by the k-nearest neighbors. 1.2 Covariance Analysis Based on the k-neighborhood relation described above, local surface properties at a point p ∈ P can be estimated using a statistical analysis of the neighboring samples. In particular, eigenanalysis of the covariance matrix of the point positions and normals of a local neighborhood yields eﬃcient algorithms for estimating normal vectors, and surface and normal variation (to be deﬁned below). Let p̄ be the centroid of the neighborhood of p, i.e., 1 pi p̄ = |Np | i∈Np The 3 × 3 covariance matrix C for the sample point p is then given as ⎤T ⎡ ⎤ ⎡ pi1 − p̄ pi1 − p̄ ⎦ ·⎣ ⎦ , ij ∈ Np ··· ··· C=⎣ pik − p̄ pik − p̄ (1.1) C describes the statistical properties of the distribution of the sample points in the neighborhood of point p by accumulating the squared distances of these points from the centroid p̄. Consider the eigenvector problem C · vl = λl · vl , l ∈ {0, 1, 2}. Since C is symmetric and positive semi-deﬁnite, all eigenvalues λl are real-valued and the eigenvectors vl form an orthogonal frame, corresponding to the principal components of the point set deﬁned by Np [Jol86]. The λl measure the variation of the pi , i ∈ Np , along the direction of the corresponding eigenvectors. The total variation, i.e., the sum of squared distances of the from the centroid is given by pi − p̄2 = λ0 + λ1 + λ2 . i∈Np Normal Estimation Assuming λ0 ≤ λ1 ≤ λ2 , it follows that the plane T (x) : (x − p̄) · v0 = 0 through p̄ minimizes the sum of squared distances to the neighbors of p [Jol86]. Thus v0 approximates the surface normal np at p, or in other words, v1 and v2 span the tangent plane at p (see Figure 1.1 (a)). A straightforward extension to this scheme uses weighted covariances, typically applying a Gaussian or polynomial weight function that drops with increasing distance to p [AA03a]. Normal Orientation A consistent orientation of the normal vectors can be computed using a method based on the Euclidean minimum spanning tree of the point cloud, as described in [HDD∗ 94]. The algorithm starts with an 1.2. COVARIANCE ANALYSIS 5 extremal point, e.g., the sample with largest z-coordinate, and orients its normal to point away from the centroid of the point cloud. The normal vector of each adjacent point in the minimum spanning tree can then be oriented based on the assumption that the angle of the normal vectors of adjacent points is less than π/2 (see Figure 1.1 (b)). If the underlying surface is orientable (note that some surface are nonorientable, e.g., the Moebius strip) and the sampling distribution is suﬃciently dense, then a consistent orientation of the normals will be obtained after all points of the point cloud have been visited. query ball v0 T p covariance ellipsoid Figure 1.1: Normal estimation (2D for illustration). (a) Computing the tangent plane using covariance analysis, (b) normal orientation using a minimum spanning tree, where the red normals have been ﬂipped, since the angle to the next adjacent oriented normal is larger then π/2. Surface Variation λ0 quantitatively describes the variation along the surface normal, i.e., estimates how strongly the points deviate from the tangent plane. The surface variation at point p in a neighborhood of size n is deﬁned as λ0 σn (p) = . λ0 + λ1 + λ2 The maximum surface variation σn (p) = 1/3 is assumed for completely isotropically distributed points, while the minimum value σn (p) = 0 indicates that all points lie in a plane. Note also that λ1 and λ2 describe the variation of the sampling distribution in the tangent plane and can thus be used to estimate local anisotropy [PGK02]. Normal Variation A similar method for local surface analysis considers the covariance matrix of the surface normals, i.e., nTi · ni . C = i∈Np Let λ0 ≤ λ1 ≤ λ2 be the eigenvalues of C with corresponding eigenvectors v0 , v1 , and v2 . As Garland discusses in [Gar99], λ2 measures the variation of the surface normals in the direction of the mean normal, while λ1 measures the maximum variation on the Gauss sphere. Thus the normal variation can be deﬁned as σn (p) = λ1 . Garland also analyzes the covariance matrix of the normal vectors in the context of diﬀerential geometry. He shows that under mild conditions on the smoothness of the surface, the eigenvectors v0 , v1 , and v2 converge to the direction of minimum curvature, maximum curvature, and mean normal, respectively, when sampling density goes to inﬁnity. Figure 1.2 compares surface variation and normal variation for the Max Planck model consisting of 139,486 points. With increasing neighborhood size, a smoothing of the variation estimates can be observed. CHAPTER 1. LOCAL SURFACE ANALYSIS 6 σ15 σ40 σ150 σ'15 σ'40 σ'150 Figure 1.2: Comparison of surface variation (left) and normal variation (right) for increasing size of the local neighborhoods. 1.3 Moving Least Squares Surfaces Given a point cloud P the goal of a point-based surface model is to deﬁne a surface S that approximates or interpolates the sample points pi ∈ P . For the algorithms deﬁned in these notes the surface model should satisfy the following requirements: • Smoothness: The surface should be smooth and diﬀerentiable, preferably in a controllable manner. This means that there should be some mechanism to adjust the smoothness depending on the intended application. • Locality: The evaluation of the surface model should be local, i.e., only points within a certain distance should inﬂuence the deﬁnition of the surface at a particular location. Locality is desirable to enable local modiﬁcations without aﬀecting the global shape of the surface. It also increases the eﬃciency of the computations. • Stability: The evaluation of the surface model should be stable and robust, even for non-uniform sampling distributions. This means that the surface model needs to be adaptive, i.e dependent on the local sampling density. Methods for interpolating or approximating functions in IRd from a discrete set of scattered data values have been studied extensively in the past. Examples include reconstruction using ﬁnite elements, radial basis functions, and moving least squares (MLS) approximation [LS86]. The latter two are advantageous because they are mesh-less, i.e., do not require a consistent tessellation of the function domain. Recently, Levin has introduced an extension of the moving least squares approximation to surfaces [Lev03]. This method can be used for computing local approximations of the surface represented by a point cloud, and to evaluate the corresponding signed distance function. The idea is to locally approximate the surface by polynomials that minimize a weighted least squares error at the data points. Since the method is solely based on Euclidean distance between sample points, no additional combinatorial structure on the point cloud is required. Given a point set P , the MLS surface S(P ) is deﬁned as the stationary set of a projection operator ΨP , i.e., S(P ) = {x ∈ IR3 |ΨP (x) = x}. The operator ΨP is deﬁned by a two-step procedure: • Compute a local reference plane H = {y ∈ IR3 |y · n − D = 0} by minimizing the weighted sum of squared distances (pi · n − D)2 φ(pi − q), (1.2) i 1.3. MOVING LEAST SQUARES SURFACES 7 where q is the orthogonal projection of x onto H. The reference plane deﬁnes a local coordinate system with q at the origin. Let (ui , vi , fi ) be the coordinates of the point pi in this coordinate system, i.e., (ui , vi ) are the parameter values in H and fi is the height of pi over H. • Compute a bivariate polynomial p̃(u, v) that minimizes (p(ui , vi ) − fi )2 φ(pi − q) (1.3) i among all p ∈ Π2m . The projection of x onto S(P ) is then given as ΨP (x) = q + p̃(0, 0) · n. Figure 1.3 illustrates the MLS projection for a curve example in 2D. The left image shows a single projection of a point x onto the MLS curve S(P ) depicted on the right. In [Lev98] Levin analyzes the smoothness and convergence rate, x ΨP(x) H S(P) q P Figure 1.3: 2D illustration of the MLS projection. which leads him to the conjecture that the smoothness of S(P ) directly depends on the smoothness of φ, i.e., if φ ∈ C k then S(P ) ∈ C k . The kernel function φ thus controls the shape of the surface 2 2 S(P ). A suitable choice for φ is the Gaussian, i.e., φ(x) = e−x /h , where h is a global scale factor that determines the kernel width. The Gaussian kernel has been used successfully in diﬀerent applications (see also [ABCO∗ 01, AA03b, FCOAS03]) and will be used in most of the algorithms discussed in these notes. The scale factor h can be understood as the characteristic scale of S(P ), i.e., features that are smaller than h will be smoothed out. In this sense the MLS projection operator implements a low-pass ﬁlter, whose ﬁlter width can be controlled by the parameter h. Figure 1.4 shows an example an MLS surface computed with diﬀerent scale factors. h=1 h=2 h=4 Figure 1.4: Smoothing eﬀect of the MLS projection. The Gaussian kernel width has been doubled for each image from left to right. 8 CHAPTER 1. LOCAL SURFACE ANALYSIS Signed Distance Function An important tool for local surface analysis is the signed distance function d+ : IR3 → IR that measures for each point x ∈ IR3 the signed distance to the surface S. Using the MLS projection an approximate distance function can be deﬁned as d(x) = x − ΨP (x). If S is closed and the normals on S are consistently oriented, e.g., always pointing outward of the surface (see Section 1.2), then the signed distance function can be formulated as d+ (x) = (x − ΨP (x)) · n, where n is the surface normal at ΨP (x) with n = 1. Computing the MLS Projection The MLS projection method deﬁned above consists of two steps: Computation of the local reference plane H and computation of the least-squares polynomial p̃ with respect to that reference plane. The former requires a non-linear optimization, since the distances used in the kernel function depend on the projection q of the point of interest x onto the unknown reference plane. Alexa et al. [ABCO∗ 01] use Powell iteration to compute D and n in Equation 1.2. A diﬀerent approach is to try and estimate directly, which determines n = (x−q)/x−q and D = q·n. This can be achieved using a Newton-type iteration based on a geometric gradient estimation as described in [PKG02]. Given the reference plane H, all points in P are projected into a local reference frame deﬁned by H. The computation of the polynomial p̃ is now a linear weighted least squares approximation, which can be computed using normal equations. For example, a cubic polynomial leads to a linear system with 10 unknowns, which can be solved using a standard linear solver. Both computation of the reference plane and the ﬁtting polynomial require order O(n) computations, since they involve the entire point cloud. However, since the value of the weight function drops quickly with distance, eﬃcient approximation schemes can be applied to signiﬁcantly reduce the complexity of the computations. For example, Alexa et al. [ABCO∗ 01] use a multi-pole scheme, where they cluster groups of points that are far away from the point of interest into a single point and use this representative in the optimization. A diﬀerent approach is to only collect points within a sphere sr of radius r centered at x, such that the weight for each point outside this sphere is so small that the inﬂuence in the least-squares optimization is negligible. For example, given an MLS scale factor h, r can be set to 3h to yield an MLS weight of less then 0.001 in Equations 1.2 and 1.3 for all points outside of sr . Adaptive MLS Surfaces So far the MLS surface approximation has been deﬁned with a ﬁxed Gaussian kernel width h. Finding a suitable h can be diﬃcult for non-uniformly sampled point clouds, however. A large scale factor will cause excessive smoothing in regions of high sampling density. Even worse, if the ﬁlter kernel is too small, only very few points will contribute signiﬁcantly to Equations 1.2 and 1.3 due to the exponential fall-oﬀ of the weight function. This can cause instabilities in the optimization because of limited precision of the computations, which lead to wrong surface approximations. Figure 1.5 shows an example of a locally uniformly sampled surface that cannot be approximated adequately using a global scale factor. The surface is deﬁned using a concentric sine wave whose wavelength and amplitude gradually increases towards the rim of the surface (see Figure 1.5 (c)). Similarly, the sampling density decreases towards the boundary as illustrated in Figures 1.5 (b) and (d). Thus the surface detail is coupled with the sampling density, i.e., in regions of high sampling density the surface exhibits high-frequency detail, whereas low-frequency detail is present where the surface is sampled less densely. Figure 1.5 (e) to (g) shows reconstructions of the central section of this surface using a regular sampling grid of 100 × 100 sample points. The Gaussian used in Figure 1.5 (e) causes substantial smoothing and leads to a signiﬁcant loss of geometric detail in the central area. In Figures 1.5 (f) and (g) the kernel width has been successively halved, which improves the approximation in the central region but leads to increasing instability towards the boundary. To cope with this problem, the MLS approximation needs to adapt to the local sampling density. In regions of high 1.3. MOVING LEAST SQUARES SURFACES 9 center (a) (b) rim center (c) rim (d) (e) (f) (g) (h) (i) (h) Figure 1.5: A surface that cannot be reconstructed using a ﬁxed kernel width. (a) surface, (b) sampling pattern, (c) vertical cross-section of surface, (d), sample spacing, (e) - (g), MLS approximation of the region shown in (h) using 10k sample points for diﬀerent kernel widths, (i) adaptive MLS approximation, (j) estimate of the local sample spacing, where blue corresponds to low values, while red indicates high values. sampling density a small Gaussian kernel should be applied to accurately reconstruct potential high geometric detail. If the sampling density is low, the kernel width needs to be bigger to improve the stability of the approximation. Given the continuous local sampling density estimate ρ of Section 1.1.1, the MLS approximation can be extended in the following way: Instead of using a ﬁxed scale factor h for all approximation, a dynamically varying scale factor can be deﬁned as h(x) = h/ρ(x), where x is the point that is to be projected onto the MLS surface. Figure 1.5 (i) shows an example of an adaptive MLS surface using the same input point cloud and sampling pattern as in (e) to (g). Observe that the high-frequency detail is faithfully recovered within the limits of the resolution of the sampling grid and that no instabilities occur at the surface boundary. 10 CHAPTER 1. LOCAL SURFACE ANALYSIS Chapter 2 Surface Simpliﬁcation Mark Pauly Point-sampled surfaces often describe complex geometric objects using millions or even billions of sample points (see for example [LPC∗ 00]). Reducing the complexity of such data sets is one of the key processing techniques for the design of scalable modeling and visualization algorithms. Surface simpliﬁcation provides a means to generate the required approximations of a given surface that use fewer sample points than the original point model. These approximations should of course resemble the original surface as closely as possible. This part of the notes describes various surface simpliﬁcation techniques for point-sampled models. Formally, the goal of point-based surface simpliﬁcation can be stated as follows: Let S be a manifold surface deﬁned by a point cloud P . Given a target sampling rate n < |P |, ﬁnd a point cloud P with |P | = n such that the distance = d(S, S ) of the corresponding surface S to the original surface S is minimal. Alternatively, a target distance can be speciﬁed and the goal is to ﬁnd the point cloud P such that d(S, S ) ≤ and |P | is minimal. These objectives require the deﬁnition of a metric d that measures the geometric distance between the original and the simpliﬁed surface. As will be described in Section 2.4.1 a discrete surface distance metric can be deﬁned using the MLS projection operator. In practice, ﬁnding the global optimum to the above problems is intractable [AS94]. Most existing surface simpliﬁcation techniques therefore use diﬀerent heuristics based on local error measures. In the following, three diﬀerent approaches will be described and analyzed [PGK02]: • Clustering methods split the point cloud into a number of disjoint subsets, each of which is replaced by one representative sample (see Section 2.1). • Iterative simpliﬁcation successively contracts point pairs in a point cloud according to a quadric error metric (Section 2.2). • Particle simulation computes new sampling positions by moving particles on the point-sampled surface according to inter-particle repulsion forces (Section 2.3). These methods are extensions and generalizations of mesh simpliﬁcation algorithms to point clouds, targeted towards densely-sampled organic shapes stemming from 3D acquisition, iso-surface extraction or sampling of implicit functions. They are less suited for surfaces that have been carefully designed in a particular surface representation, such as low-resolution polygonal CAD data. Furthermore, the goal is to design algorithms that are general in the sense that they do not require any knowledge of the speciﬁc source of the data. For certain applications this additional knowledge could be exploited to design more eﬀective simpliﬁcation algorithms, but this would also limit the applicability of these methods. 11 CHAPTER 2. SURFACE SIMPLIFICATION 12 The algorithms described in this chapter diﬀer in a number of aspects such as quality of the generated surfaces, computational eﬃciency and memory overhead. These features are discussed in a comparative analysis in Section 2.4. The purpose of this analysis is to give potential users of pointbased surface simpliﬁcation suitable guidance for choosing the right method for their speciﬁc application. Real-time applications, for instance, will put particular emphasis on eﬃciency and low memory footprint. Methods for creating surface hierarchies favor speciﬁc sampling patterns (e.g., [Tur01]), while visualization applications require accurate preservation of appearance attributes, such as color or material properties. Earlier methods for simpliﬁcation of point-sampled models have been introduced by Alexa et al. [ABCO∗ 01] and Linsen [Lin01]. These algorithms create a simpliﬁed point cloud that is a true subset of the original point set by ordering iterative point removal operations according to a surface error metric. While both papers report good results for reducing redundancy in point sets, pure subsampling unnecessarily restricts potential sampling positions, which can lead to aliasing artifacts and uneven sampling distributions. To alleviate these problems, the algorithms described here re-sample the input surface and implicitly apply a low-pass ﬁlter (e.g., clustering methods perform a local averaging step to compute the cluster’s centroid). In [PG01], Pauly and Gross introduced a re-sampling strategy based on Fourier theory. They split the model surface into a set of patches that are re-sampled individually using a spectral decomposition. This method directly applies signal processing theory to point-sampled geometry, yielding a fast and versatile point cloud decimation method. Potential problems arise due to the dependency on the speciﬁc patch layout and diﬃculties in controlling the target model size by specifying spectral error bounds. 2.1 Clustering Clustering methods have been used in many computer graphics applications to reduce the complexity of 3D objects. Rossignac and Borrel, for example, used vertex clustering to obtain multi-resolution approximations of complex polygonal models for fast rendering [RB93]. The standard strategy is to subdivide the model’s bounding box into grid cells and replace all sample points that fall into the same cell by a common representative. This volumetric approach has some drawbacks, however. By using a grid of ﬁxed size this method cannot adapt to non-uniformities in the sampling distribution. Furthermore, volumetric clustering easily joins unconnected parts of a surface, if the grid cells are too large. To alleviate these shortcomings, surface-based clustering techniques can be applied, where clusters are build by collecting neighboring samples while regarding local sampling density. Two general approaches for building clusters will be described in these notes. An incremental method, where clusters are created by region-growing, and a hierarchical approach that splits the point cloud into smaller subsets in a top-down manner [SG01]. Both methods create a set of clusters {Ci }, such that for every point pj ∈ P there exists a unique cluster Ci with pj ∈ Ci . The simpliﬁed point cloud is then obtained by replacing each cluster Ci by a representative sample, typically its centroid given as 1 p̄i = pj . (2.1) |Ci | j∈Ci 2.1.1 Incremental Clustering Starting from a random seed point pj ∈ P , a cluster C0 is built by successively adding nearest neighbors. This incremental region-growing is terminated when the size of the cluster reaches a maximum bound. Additionally, the maximum allowed variation σn of each cluster can be restricted (see Section 1.2). This results in a curvature-adaptive clustering method, where more and smaller clusters are created in regions of high surface variation. The next cluster C1 is then build by starting the incremental growth with a new seed chosen from the neighbors of C0 and excluding all points of from 2.1. CLUSTERING 13 C0 the region-growing. This process is terminated when all sample points of P have been assigned to a cluster in {Ci }. Due to fragmentation, this method creates many clusters that did not reach the maximum size or variation bound, but whose incremental growth was restricted by adjacent clusters. To obtain a more even distribution of clusters, sample points of all clusters that did not reach a minimum size and variation bound (typically half the values of the corresponding maximum bounds) are distributed to neighboring clusters (see Figure 2.1). Note that this potentially increases the size and variation of the clusters beyond the user-speciﬁed maxima. Figure 2.2 illustrates incremental clustering, where oriented circular splats are used to indicate the sampling distribution. C2 C0 C1 Figure 2.1: Fragmentation of incremental clustering. Gray dots correspond to sample points that are assigned to clusters that reached the necessary size and variation bounds. All other points are stray samples (blue dots) and will be attached to the cluster with closest centroid. Figure 2.2: Uniform incremental clustering: The left image illustrates the corresponding clusters on the original point set (296,850 points), while the right image shows the simpliﬁed point set (2,413 points). CHAPTER 2. SURFACE SIMPLIFICATION 14 2.1.2 Hierarchical Clustering An alternative method for computing the set of clusters recursively splits the point cloud using a binary space partition. The point cloud P is split, if • the size |P | is larger than the user speciﬁed maximum cluster size nmax , or • the variation σn (P ) is above a maximum threshold σmax . The split plane is deﬁned by the centroid of P and the eigenvector v2 of the covariance matrix of with largest corresponding eigenvector (see also Figure 1.1). Hence the point cloud is always split along the direction of greatest variation [SG01]. If the splitting criterion is not fulﬁlled, the point cloud P becomes a cluster Ci . As shown in Figure 2.3, hierarchical clustering builds a binary tree, where each leaf of the tree corresponds to a cluster. A straightforward extension to the recursive scheme uses a priority queue to order the splitting operations [BW00, SG01]. While this leads to a signiﬁcant increase in computation time, it allows direct control over the number of generated samples, which is diﬃcult to achieve by specifying nmax and σmax only. Figure 2.4 illustrates adaptive hierarchical clustering. split plane covariance ellipsoid v2 centroid leaf node = cluster Figure 2.3: Three intermediate steps of the hierarchical clustering algorithm. (a) 2D sketch, (b) uniform hierarchical clustering for the Max Planck model. 2.2 Iterative Simpliﬁcation A diﬀerent strategy for point-based surface simpliﬁcation iteratively reduces the number of points using an atomic decimation operator. This approach is very similar to mesh-based simpliﬁcation methods for creating progressive meshes [Hop96]. Decimation operations are usually arranged in a priority queue according to an error metric that quantiﬁes the error caused by the decimation. The iteration is then performed in such a way that the decimation operation causing the smallest error is applied 2.3. PARTICLE SIMULATION 15 Figure 2.4: Adaptive hierarchical clustering: The left image illustrates the clusters on the original point set, while the right image shows the simpliﬁed point set (1,831 points). The size of the splats on the right image is proportional to the corresponding cluster size. ﬁrst. Earlier work [ABCO∗ 01, Lin01] uses simple point removal, i.e., points are iteratively removed from the point cloud, resulting in a simpliﬁed point cloud that is a true subset of the original point set. As discussed above, this can lead to undesirable artifacts, which can be avoided by using point-pair contraction instead of point removal. This extension of the common edge collapse operator replaces two points p1 and p2 by a new point p̄ implicitly applying a low-pass ﬁlter by computing a weighted average of the contracted point pair. To rate the cost of a contraction operation, an adaptation of the quadric error metric is used as presented for polygonal meshes in [GH97]. The idea there is to approximate the surface locally by a set of tangent planes and to estimate the geometric deviation of a mesh vertex v from the surface by the sum of the squared distances to these planes. The error quadrics for each vertex v are initialized with a set of planes deﬁned by the triangles around that vertex and can be represented by a symmetric 4 × 4 matrix Qv . The quality of the collapse (v1 , v2 ) → v̄ is then rated according to the minimum of the error functional Qv̄ = Qv1 + Qv2 . In order to adapt this technique to the decimation of unstructured point clouds, manifold surface connectivity is replaced by the k-nearest neighbor relation (see Section 1.1). The error quadrics for every point sample p are initialized by estimating a tangent plane Ei for every edge that connects p with one of its neighbors pi . This tangent plane is spanned by the vectors ei = p − pi and bi = ei × n, where n is the estimated normal vector at p. After this initialization the point cloud decimation works exactly like mesh decimation with the point p̄ inheriting the neighborhoods of its ancestors and and being assigned the error functional Qp̄ = Qp1 + Qp2 . Figure 2.5 shows an example of a simpliﬁed point cloud created by iterative point-pair contraction. 2.3 Particle Simulation In [Tur92], Turk introduced a method for re-sampling polygonal surfaces using particle simulation. The desired number of particles is randomly spread across the surface and their position is equalized using a point repulsion algorithm. Point movement is restricted to the surface deﬁned by the individual polygons to ensure an accurate approximation of the original surface. Turk also included a curvature estimation method to concentrate more samples in regions of high curvature. Finally, the new vertices CHAPTER 2. SURFACE SIMPLIFICATION 16 Figure 2.5: Iterative simpliﬁcation of the Max Planck model from 296,850 (left) to 2,000 sample points (middle). The right image shows all remaining potential point-pair contractions indicated as an edge between two points. Note that these edges do not necessarily form a consistent triangulation of the surface. are re-triangulated yielding the re-sampled triangle mesh. This scheme can easily be adapted to pointsampled geometry. 2.3.1 Spreading Particles Turk initializes the particle simulation by randomly distributing points on the surface. Since a uniform initial distribution is crucial for fast convergence, this random choice is weighted according to the area of the polygons. For point-based models, this area measure can be replaced by a density estimate (see Section 1.1.1). Thus by placing more samples in regions of lower sampling density (which correspond to large triangles in the polygonal setting), uniformity of the initial sample distribution can ensured. 2.3.2 Repulsion For repulsion the same linear force term is used as in [Tur92], because its radius of inﬂuence is ﬁnite, i.e., the force vectors can be computed very eﬃciently as, Fi (p) = k(r − p − pi ) · (p − pi ), (2.2) where Fi (p) is the force exerted on particle p due to particle pi , k is a force constant and r is the repulsion radius. The total force exerted on p is then given as F (p) = Fi (p), (2.3) i∈Np where Np is the neighborhood of p with radius r. Using a 3D grid data structure, this neighborhood can be computed eﬃciently in constant time. 2.3.3 Projection In Turk’s method, displaced particles are projected onto the closest triangle to prevent the particles from drifting away from the surface. Since no explicit surface representation is available, the MLS projection operator (see Section 1.3) is applied to keep the particles on the surface. However, applying 2.4. COMPARISON 17 this projection every time a particle position is altered is computationally too expensive. Therefore, a diﬀerent approach has been used: A particle p is kept close to the surface by simply projecting it onto the tangent plane of the point p of the original point cloud that is closest to p. The full moving least squares projection is only applied at the end of the simulation, which alters the particle positions only slightly and does not change the sampling distribution noticeably. 2.3.4 Adaptive Simulation Using the variation estimate of Section 1.2, more points can be concentrated in regions of high curvature by scaling their repulsion radius with the inverse of the variation σn . It is also important to adapt the initial spreading of particles accordingly to ensure fast convergence. This can be done by replacing the density estimate ρ by ρ · σn . Figure 2.6 gives an example of an adaptive particle simulation. Figure 2.6: Simpliﬁcation by adaptive particle simulation. The left image shows the repulsion radius on the original model determined from the surface variation estimate. Blue indicates a large radius, while red indicates small radius. On the right, a simpliﬁed model consisting of 3,000 points is shown, where the size of the splats is proportional to the repelling force of the corresponding particle. Scaling of the repulsion radius also provides an easy means for user-controlled surface re-sampling. For this purpose a painting tool can be used (see also Chapter 4) that allows the user to directly paint the desired repulsion radius onto the surface and thus control the sampling distribution of the re-sampled surface. As illustrated in Figure 2.7, particle simulation automatically creates smooth transitions between areas of diﬀerent sampling density. 2.4 Comparison The previous sections have introduced diﬀerent algorithms for point-based surface simpliﬁcation. As mentioned before, none of these methods attempts to ﬁnd an optimal point distribution with respect to a global distance metric, but rather uses some built-in heuristics to approximate such an optimum: • Clustering methods try to partition the point cloud into clusters of equal size and/or surface variation, assuming that each cluster describes an equally important part of the surface. • Iterative simpliﬁcation optimizes locally according to the quadric error metric. CHAPTER 2. SURFACE SIMPLIFICATION 18 Figure 2.7: Uniform and user-controlled particle simulation. The model in the middle has been simpliﬁed to 2,000 points in the left image using uniform repulsion forces. On the right, the repulsion radius has been scaled down to 10% around the eye, leading to a higher concentration of samples in this region. • Particle simulation is based on the assumption that a minimum of the potential energy of the particles minimizes the distance between original and re-sampled surface. Since these heuristics are fundamentally diﬀerent, the surfaces generated by these algorithms will diﬀer signiﬁcantly in general. Thus, to evaluate and compare the quality of the simpliﬁed surfaces, some generic technique for measuring the geometric distance between two point-sampled surfaces is required. This distance measure should be general in the sense that no additional knowledge about the speciﬁc method used to generate the simpliﬁed surface should be required. Further aspects in the evaluation of the various simpliﬁcation algorithms include sampling distribution of the simpliﬁed model, time and space eﬃciency, and implementation issues. 2.4.1 Surface Error Assume that two point clouds P and P are given, which represent two surfaces S and S , respectively. The distance, or error, between these two surfaces is measured using a sampling approach similar to the method applied in the Metro tool [CRS]. Let Q be a set of points on S and let d(q, S ) = minx∈S d(q, x) be the minimum distance of a point q ∈ Q to the surface S . Then two error measures can be deﬁned: • Maximum error: ∆max (S, S ) = maxq∈Q d(q, S ) (2.4) The maximum error approximates the two-sided Hausdorﬀ distance of the two surfaces. Note that the surface-based approach is crucial for meaningful error estimates, as the Hausdorﬀ distance of the two point sets P and P does not adequately measure the distance between S and S . • Average error: ∆avg (S, S ) = 1 d(q, S ) |Q| (2.5) q∈Q The average error approximates the area-weighted integral of the point-to-surfaces distances. 2.4. COMPARISON 19 The point set Q is created using the uniform particle simulation of Section 2.3. This allows the user to control the accuracy of the estimates 2.4 and 2.5 by specifying the number of points in Q. To obtain a visual error estimate, the sample points of Q can be color-coded according to the point-to-surface distance and rendered using a standard point rendering technique (see Figures 2.9 and 2.10). d(q, S ) is calculated using the MLS projection operator Ψ with linear basis functions (see Section 1.3). Eﬀectively, Ψ approximates the closest point q ∈ S such that q = q + d · n for a q ∈ Q, where n is the surface normal at q and d is the distance between q and q (see Figure 2.8). Thus the point-to-surface distance d(q, S ) is given as d = q − q . q p S p' } d q' S' Figure 2.8: Measuring the distance between two surfaces S(red curve) and S (black curve) represented by two point sets P (red dots) and P (black dots). P is up-sampled to Q (blue dots) and for each q ∈ Q a base point q ∈ S is found (green dots). The point-to-surface distance d(q, S ) is then equal to d = q − q . Figure 2.9: Measuring surface error. From left to right: original surface, simpliﬁed point cloud, surface obtained by up-sampling the simpliﬁed point cloud, color-coded error, where blue corresponds to a small error, while red indicates a large error. Figure 2.10 shows visual and quantitative error estimates (scaled according to the objects bounding box diagonal) for the David model that has been simpliﬁed from 2,000,606 points to 5,000 points. Uniform incremental clustering has the highest average error. Since all clusters consist of roughly the same number of sample points, most of the error is concentrated in regions of high curvature. Adaptive hierarchical clustering performs slightly better, in particular in the geometrically complex regions of the hair. Iterative simpliﬁcation and particle simulation provide lower average error and distribute the error more evenly across the surface. In general the iterative simpliﬁcation method using quadric error metrics has been found to produce the lowest average surface error. Clustering methods CHAPTER 2. SURFACE SIMPLIFICATION 20 perform worse with respect to surface error, because they do not have the ﬁne-grain adaptivity of the iterative simpliﬁcation and particle simulation methods. For example, consider the hierarchical clustering algorithm as illustrated in Figure 2.3. The split planes generated in the earlier stages of the recursion are propagated down to all subsequent levels. This means that once a top-level split plane has been chosen, the method cannot adapt to local variations across that split plane, which leads to increased surface error. ∆avg = 6.32 * 10 -4 ∆max = 0.0049 (a) uniform incremental clustering ∆avg = 6.14 * 10 -4 ∆max = 0.0046 (b) adaptive hierarchical clustering ∆avg = 5.43 * 10 -4 ∆max = 0.0052 (c) iterative simplification ∆avg = 5.69 * 10 -4 ∆max = 0.0061 (d) particle simulation Figure 2.10: Surface Error for Michelangelos David simpliﬁed from 2,000,606 points to 5,000 points. 2.4.2 Sampling Distribution Apart from the geometric error, the distribution of samples within the surface can be an important aspect for certain applications. As mentioned before, all simpliﬁcation algorithms described here create a point cloud that is in general not a subset of the original sample set. Where this is required, methods such as those presented by Alexa et al. [ABCO∗ 01] or Linsen [Lin01] are preferable. For clustering methods the sampling distribution in the ﬁnal model is closely linked to the sampling distribution of the input model. In some applications this might be desirable, e.g., where the initial sampling pattern carries some semantic information, such as in geological models. Other applications, e.g., pyramid algorithms for multi-level smoothing [KCVS98] or texture synthesis [Tur01], require uniform sampling distributions, even for highly non-uniformly sampled input models. Here non-adaptive particle simulation is most suitable, as it distributes sample points uniformly and independently of the sampling distribution of the underlying surface. As illustrated in Figure 2.7, particle simulation also provides a very easy mechanism for locally controlling the sampling density by scaling the repulsion radius accordingly. While similar eﬀects can be achieved for iterative simpliﬁcation by penalizing certain point-pair contractions, particle simulation oﬀers much more intuitive control. Note that none of the surface simpliﬁcation methods described above gives any guarantees that the resulting point cloud satisﬁes speciﬁc sampling criteria (see eg [ABK98]). It is rather left to the application to specify a suitable target sampling rate. 2.4.3 Computational Eﬀort Figure 2.11 shows computation times for the diﬀerent simpliﬁcation methods both as a function of target model size and input model size. Due to the simple algorithmic structure, clustering methods are by far the fastest simpliﬁcation techniques discussed in these notes. Iterative simpliﬁcation has a relatively long pre-computing phase, where initial contraction candidates and corresponding error quadrics are determined and the priority queue is set up. The simple additive update rule of the quadric metric (see Section 2.2) make the simpliﬁcation itself very eﬃcient, however. In the current 2.4. COMPARISON 21 implementation particle simulation is the slowest simpliﬁcation technique for large target model sizes, mainly due to slow convergence of the relaxation step. A possible improvement is the hierarchical approach introduced in [WH94]]. The algorithm would start with a small number of particles and relax until the particle positions have reached equilibrium. Then particles are split, their repulsion radius is adapted and relaxation continues. This scheme can be repeated until the desired number of particles is obtained. Execution time (secs.) Execution time (secs.) Incremental Clustering 70 60 Hierarchical Clustering 500 Incremental Clustering Iterative Simplification 450 Hierarchical Clustering Particle Simulation 400 Iterative Simplification 350 Particle Simulation 50 St.Matthew 300 40 250 200 20 150 David 50 0 0 180 Sa Igenta Isis a 100 10 Dr ag on 30 160 140 120 100 80 60 40 20 0 0 500 1000 1500 2000 simplified model size (*1000) (a) 2500 3000 3500 input model size (*1000) (b) Figure 2.11: Execution times for simpliﬁcation, measured on a Pentium 4 (1.8GHz) with 1Gb of main memory: (a) as a function of target model size for the dragon model (435,545 points), (b) as a function of input model size for a simpliﬁcation to 1%. It is interesting to note that for incremental clustering and iterative simpliﬁcation the execution time increases with decreasing target model size, while hierarchical clustering and particle simulation are more eﬃcient the smaller the target models. Thus the latter are more suitable for real-time applications where the fast creation of coarse model approximations is crucial. 2.4.4 Memory Requirements and Data Structures Currently all simpliﬁcation methods discussed in these notes have been implemented in-core, i.e., require the complete input model as well as the simpliﬁed point cloud to reside in main memory. For incremental clustering a balanced kd-tree is used for fast nearest-neighbor queries, which can be implemented eﬃciently as an array [Sed98], requiring 4 · n bytes, where n is the size of the input model. Hierarchical clustering builds a BSP tree, where each leaf node corresponds to a cluster. Since the tree is built by re-ordering the sample points, each node only needs to store the start and end index in the array of sample points and no additional pointers are required. Thus this maximum number of additional bytes is 2 · 2 · 4 · m, where m is the size of the simpliﬁed model. Iterative simpliﬁcation requires 96 bytes per point contraction candidate, 80 of which are used for storing the error quadric (ﬂoating point numbers are stored in double precision, since single precision ﬂoats lead to numerical instabilities). Assuming six initial neighbors for each sample point, this amounts to 6/2 · 96 · n bytes. Particle simulation uses a 3D grid data structure with bucketing to accelerate the nearest neighbor queries, since a static kd-tree is not suitable for dynamically changing particle positions. This requires a maximum of 4 · (n + k) bytes, where k is the resolution of the grid. Thus incremental clustering, iterative simpliﬁcation and particle simulation need additional storage that is linearly proportional to the number of input points, while the storage overhead for hierarchical clustering depends only on the target model size. CHAPTER 2. SURFACE SIMPLIFICATION 22 2.4.5 Comparison to Mesh Simpliﬁcation Figure 2.12 shows a comparison of point-based simpliﬁcation and simpliﬁcation for polygonal meshes. In (a), the initial point cloud is simpliﬁed from 134,345 to 5,000 points using the iterative simpliﬁcation method of Section 2.2. The resulting point cloud has then been triangulated using the surface reconstruction method of [GJ02]. In (b), the input point cloud has ﬁrst been triangulated and the resulting polygonal surface has then been simpliﬁed using the mesh simpliﬁcation tool QSlim [GH97]. Both methods produce similar results in terms of surface error and both simpliﬁcation processes take approximately the same time (approx. 3.5 seconds). However, creating the triangle mesh from the simpliﬁed point cloud took 2.45 seconds in (a), while in (b) reconstruction time for the input point cloud was 112.8 seconds. Thus when given a large unstructured point cloud, it is much more eﬃcient to ﬁrst do the simpliﬁcation on the point data and then reconstruct a mesh (if desired) than to ﬁrst apply a reconstruction method and then simplify the triangulated surface. This illustrates that point-based simpliﬁcation methods can be very useful when dealing with large geometric models stemming from 3D acquisition. (a) point-based simplification (b) mesh-based simplification Figure 2.12: Comparison of point-based (a) and mesh-based (b) surface simpliﬁcation. Error images are shown on the left, ﬁnal triangulated surfaces on the right. Chapter 3 Shape Modeling Mark Pauly Modeling the shape of 3D objects is one of the fundamental techniques in digital geometry processing. In this chapter two fundamental modeling approaches for point-sampled geometry will be discussed: Boolean operations and free-form deformation [PKKG03]. While the former are concerned with building complex objects by combining simpler shapes, the latter deﬁnes a continuous deformation ﬁeld in space to smoothly deform a given surface. Boolean operations are most easily deﬁned on implicit surface representations, since the required inside-outside classiﬁcation can be directly evaluated on the underlying scalar ﬁeld. On the other hand, free-form deformation is a very intuitive modeling paradigm for explicit surface representations. For example, mesh vertices or NURBS control points can be directly displaced according to the deformation ﬁeld. For point-based representations, the hybrid structure of the MLS surface model can be exploited to integrate these two modeling approaches into a uniﬁed shape modeling framework. Boolean operations can utilize the approximated signed distance function deﬁned by the MLS projection (see Section 1.3) for inside-outside classiﬁcation, while free-form deformations operate directly on the point samples. 3.1 Boolean Operations A common approach in geometric modeling is to build complex objects by combining simpler shapes using boolean operations (see Figure 3.1). In constructive solid geometry (CSG) objects are deﬁned using a binary tree, where each node corresponds to a union, intersection, or diﬀerence operation and each leaf stores a base shape. Operations such as ray-tracing, for example, are then implemented by traversing this tree structure. More commonly, surfaces are deﬁned as boundary representations (BReps) of solids. Here boolean operations have to be evaluated explicitly, which requires an algorithm for intersecting two surfaces. Computing such a surface-surface intersection can be quite involved, however, in particular for higher order surfaces (see for example [KM97]). As will be demonstrated below, the MLS projection operator can be used both for inside/outside classiﬁcation as well as for explicitly sampling the intersection curve. The goal is to perform a boolean operation on two orientable, closed surfaces S1 and S2 that are represented by two point clouds P1 and P2 , to obtain a new point cloud P3 that deﬁnes the resulting surface S3 . P3 consists of two subsets Q1 ⊆ P1 and Q2 ⊆ P2 plus a set of newly generated sample points that explicitly represent the intersection curves. Thus in order to perform a boolean operation for point-sampled geometry, the following techniques are required: • a classiﬁcation predicate to determine the two sets Q1 and Q2 , • an algorithm to ﬁnd samples on the intersection curve, and 23 CHAPTER 3. SHAPE MODELING 24 • a rendering method that allows to display crisp features curves using point primitives. A+B A-B A*B Figure 3.1: Boolean operations of a sphere A and a cylinder B. The bottom row illustrates the sampling distribution. 3.1.1 Classiﬁcation The goal of the classiﬁcation stage is to determine which p ∈ P1 are inside or outside the volume enclosed by the surface S2 and vice versa. For this purpose a classiﬁcation predicate ΩP is deﬁned such that for x ∈ IR3 1 x∈V ΩP (x) = (3.1) 0 x∈ / V, where V is the volume bounded by the MLS surface S represented by the point cloud P . Let y ∈ S be the closest point on S from x. It is well-known from diﬀerential geometry that, if S is continuous and twice diﬀerentiable, the vector x − y is aligned with the surface normal ny at y [dC76]. If surface normals are consistently oriented to point outwards of the surface, then (x − y) · ny > 0, if and only if x ∈ / V . Since only a discrete sample P of the surface is given, the closest point y on S is replaced by the closest point p ∈ P . Thus x is classiﬁed as outside,if (x − p) · np > 0, i.e., if the angle between x − p and the normal np at is less than π/2 (see Figure 3.2, left image). This discrete test yields the correct inside/outside classiﬁcation of the point x, if the distance x − p is larger than the local sample spacing ηp at p. If x is extremely close to the surface, the classiﬁcation could fail, as illustrated in the right image of Figure 3.2. In this case the exact closest point y ∈ S is approximated using the MLS projection. Since for classiﬁcation only an inside/outside test is of interest, the performance can be signiﬁcantly improved by exploiting local coherence: ΩP (x) = ΩP (x ) for all points x that lie in the sphere around x with radius x − p − ηp . Thus the number of closest point queries and MLS projections can be reduced drastically, in practice up to 90 percent. Given the classiﬁcation predicate Ω, the subsets Q1 and Q2 can be computed as shown in Table 3.1. As Figure 3.3 illustrates, the resulting inside/outside classiﬁcation is very robust and easily handles complex, non-convex surfaces. Observe that boolean operations can create a large number of disconnected components, i.e., can lead to a signiﬁcant change in genus. 3.1. BOOLEAN OPERATIONS x 25 ny np ny p y x y p >90o np Figure 3.2: Inside/outside test. For x very close to the surface, the closest point p ∈ P can yield a false classiﬁcation (right image). In this case, x is classiﬁed according to its MLS projection y. S1 ∪ S2 S1 ∩ S2 S1 − S2 S2 − S1 Q1 {p ∈ P1 ||ΩP2 (p) = 0} {p ∈ P1 ||ΩP2 (p) = 1} {p ∈ P1 ||ΩP2 (p) = 0} {p ∈ P1 ||ΩP2 (p) = 1} Q2 {p ∈ P2 ||ΩP1 (p) = 0} {p ∈ P2 ||ΩP1 (p) = 1} {p ∈ P2 ||ΩP1 (p) = 1} {p ∈ P2 ||ΩP1 (p) = 0} Table 3.1: Classiﬁcation for Boolean operations. A+B A*B A-B B-A Figure 3.3: Boolean operations of a blue dragon (A) and a white dragon (B). 3.1.2 Intersection Curves Taking the union of Q1 and Q2 will typically not produce a point cloud that accurately describes the surface S3 , since the intersection curve of the two MLS surfaces S1 and S2 is not represented adequately. Therefore, a set of sample points that lie on the intersection curve is explicitly computed and added to Q1 ∪ Q2 , to obtain the point cloud P3 . First, all points in Q1 and Q2 are found that are close to the CHAPTER 3. SHAPE MODELING 26 intersection curve by evaluating the distance function induced by the MLS projection operator. From all closest pairs (q1 ∈ Q1 , q2 ∈ Q2 ) of these points a point q on the intersection curve is computed using a Newton-type iteration. This is done as follows (see Figure 3.4 (a-d)): Let r be the point on the intersection line of the two tangent planes of q1 and q2 that is closest to both points, i.e., that minimizes the distance r − q1 + r − q2 . r is the ﬁrst approximation of q and can now be projected onto S1 and S2 to obtain two new starting points q1 and q2 for the iteration. This procedure can be repeated iteratively until the points q1 and q2 converge to a point q on the intersection curve. Due to the quadratic convergence of the Newton iteration, this typically requires less than three iterations. The sampling density estimation of Section 1.1 is used to detect whether the sampling resolution of intersection curve Q1 Q2 S1 r q1 q2 (a) q1' S2 (b) r' q2' (c) (d) Figure 3.4: Sampling the intersection curve. (a) closest pairs of points in Q1 and Q2 , (b) ﬁrst estimate r, (c) re-projection, (d) second estimate r . the two input surfaces diﬀers signiﬁcantly in the vicinity of the intersection curve. To avoid a sharp discontinuity in sampling density, the coarser model is up-sampled in this area to match the sampling density of the ﬁner model, using the dynamic sampling method of Section 3.2.2. Note that the above Newton scheme also provides an easy mechanism for adaptively reﬁning the intersection curve. A simple subdivision rule is evaluated to create a new starting point for the Newton iteration, e.g., the average of two adjacent points on the curve. Applying the iteration then yields a new sample on the intersection curve (see Figure 3.5). (a) (b) (c) Figure 3.5: Adaptive reﬁnement. (a) original intersection curve, (b) new point inserted in region of high curvature, (c) ﬁnal, adaptively sampled intersection curve. 3.1. BOOLEAN OPERATIONS 3.1.3 27 Rendering Sharp Creases The accurate display of the intersection curves requires a rendering technique that can handle sharp creases and corners. For this purpose an extension of the surface splatting technique presented in [ZPvG01] is used [ZRB∗ 04]. In this method, each sample point is represented by a surfel, an oriented elliptical splat that is projected onto the screen to reconstruct the surface in image space. A point on the intersection curve can now be represented by two surfels that share the same center, but whose normals stem from either one of the two input surfaces. During scan-conversion, each of these surfels is then clipped against the plane deﬁned by the other to obtain a piecewise linear approximation of the intersection curve in screen space (see Figure 3.6). This concept can easily be generalized to handle corners as shown in Figure 3.6 (e). (a) (b) (e) (c) (d) Figure 3.6: Rendering the intersection curve. (a) mutual clipping of two surfels on the intersection curve, (b) boolean diﬀerences on the bunny model, (c) zoom of the intersection curves, (d) sampling distribution, where samples on the intersection curve are rendered using two red half ellipses, (e) an example of a corner. Figure 3.7 shows an example of a diﬃcult boolean operation of two identical cylinders that creates two singularities. While the classiﬁcation and intersection curve sampling work ﬁne, the rendering method produces artifacts. This is due to numerical instabilities, since the clipping planes of two corresponding surfels are almost parallel. However, such cases are rare in typical computer graphics applications, e.g., digital character design. As such, the algorithms for boolean operations are less suited for industrial manufacturing applications, where robust handling of degenerated cases is of primary concern. 3.1.4 Particle-Based Blending As illustrated in Figure 3.1, boolean operations typically produce sharp intersections. In some applications it is more desirable to create a smooth blend between the two combined surface parts. To smooth out the sharp creases created by boolean operations an adaptation of oriented particles [ST92] can be used. The idea is to deﬁne inter-particle potentials in such a way that the minimum of the global potential function yields a smooth surface that minimizes curvature. Summing up these potentials yields a particle’s total potential energy. From this potential energy one can derive the positional and rotational forces that are exerted on each particle and compute its path of motion under these forces. CHAPTER 3. SHAPE MODELING 28 Figure 3.7: A diﬃcult boolean diﬀerence operation that creates two singularities. Additionally, an inter-particle repulsion force is applied to equalize the particle distribution (see also Section 2.3). All forces are scaled with a smooth fall-oﬀ function that measures the distance to the intersection curve to conﬁne the particle simulation to a small area around the intersection curve without aﬀecting other parts of the surface. Figure 3.8 shows the particle-based blending for the intersection of two planes, where the degree of smoothness can be controlled by the number of iterations of the simulation. In Figure 3.9, a more complex blending operation is shown. A union operation of three tori has created numerous sharp intersection curves as shown in (a). These can be blended simultaneously as illustrated in (b) using the particle simulation described above. The same blending technique can of course also be applied to the intersection and diﬀerence operations described in Section 3.1. 3.2 Free-Form Deformation Apart from composition of surfaces using boolean operations, many shape design applications require the capability to modify objects using smooth deformations [Bar84, SP86]. These include bending, twisting, stretching, and compressing of the model surface. For this purpose a point-based free-form deformation tool is described that allows the user to interactively deform a surface by specifying a smooth deformation ﬁeld [PKKG03]. The user ﬁrst deﬁnes a deformable region χd ⊂ S on the model surface and marks parts of this region as a control handle. The surface can then be modiﬁed by pushing, pulling or twisting this handle. These user interactions are translated into a continuous tensor-ﬁeld, which for each point in the deformable region deﬁnes a translatory and rotational motion under which the surface deforms. The tensor-ﬁeld is based on a continuously varying scale parameter t ∈ [0, 1] that measures the relative distance of a point from the handle. The closer a point is to the handle, the stronger will the deformation be for that point. More precisely, let χ1 ⊂ χd be the handle, also called one-region, and χ0 = S − χd the zero-region, i.e., all points that are not part of the deformable region. For both zero- and one-region distance measures d0 and d1 ,respectively, are deﬁned as 0 p ∈ χj (3.2) dj (p) = minq∈χj p − q p ∈ / χj for j = 0, 1. From these distance measures the scale parameter t is computed as t = β(d0 (p)/(d0 (p) + d1 (p))), where β : [0, 1] → [0, 1] is a continuous blending function with β(0) = 0 and β(1) = 1. Thus t = 0 for p ∈ χ0 and t = 1 for p ∈ χ1 . Using this scale parameter, the position of a point p ∈ χd after the 3.2. FREE-FORM DEFORMATION 29 initial configuration 10 iterations 20 iterations 50 iterations 100 iterations 200 iterations Figure 3.8: Particle simulation to blend two intersecting planes. Gray particles participate in the simulation, blue points indicate the ﬁxed boundary. Figure 3.9: Boolean union of three tori. Left: Reconstruction with sharp feature curves, right: Feature curves have been blended using particle simulation. deformation is determined as p = F (p, t), where F is a deformation function composed of a translatory and a rotational part. The deformation function can be written as F (p, t) = FT (p, t) + FR (p, t), where • FT (p, t) = p + t · v with v a translation vector and • FR (p, t) = R(a, t · α) · p, where R(a, α) is the matrix that speciﬁes a rotation around axis a with angle α. Figure 3.10 shows a translatory deformation of a plane where the translation vector v is equal to the plane normal. This ﬁgure also illustrates the eﬀect of diﬀerent choices of the blending function CHAPTER 3. SHAPE MODELING 30 β. In Figure 3.11, two rotational deformations of a cylinder are shown, while a combination of both translatory and rotational deformations is illustrated in Figure 3.16. To perform a free-form deformation the user only needs to select the zero- and one-regions and choose an appropriate blending function. She can then interactively deform the surface by displacing the handle with a mouse or trackball device, similar to [KCVS98]. This gives the method great ﬂexibility for handling a wide class of free-form deformations, while still providing a simple and intuitive user interface. The deformable region and the handle can be speciﬁed using a simple paint tool that allows the user to mark points on the surface by drawing lines, circles, rectangles, etc. and applying ﬂood ﬁlling and erasure. The system also supports pre-deﬁned and user-editable selection stencils, which can be used to create embossing eﬀects (Figure 3.12). 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1 1 0.8 0.6 0.4 0.2 0 1 0.8 0.6 0.4 0.2 0 Figure 3.10: Deformations of a plane for three diﬀerent blending functions. Left: Blending function, middle: Color-coded scale parameter, where blue indicates the zero region (t = 0) and red the oneregion (t = 1 ), right: Final textured surface. 3.2.1 Topology Control An important issue in shape design using free-form deformation is the handling of self-intersections. During deformation, parts of the deformable region can intersect other parts of the surface, which leads to an inconsistent surface representation. A solution to this problem requires a method for detecting and resolving such collisions. 3.2. FREE-FORM DEFORMATION 31 a1 a2 (a) (b) (c) (d) Figure 3.11: Rotational deformations of a cylinder. (a) original, (b) color-coded scale parameter, (c) rotation around axis a1 , (d) rotation around axis a2 . Figure 3.12: Embossing eﬀect. The Siggraph label on the top left has been converted to a selection stencil using simple image processing tools. This stencil can then be mapped to a surface, where blue color corresponds to the zero-region and red to the one-region. Subsequent deformation yields an embossing eﬀect as shown on the right. Collision Detection Similar to boolean operations, this requires an inside/outside classiﬁcation to determine which parts of the surface have penetrated others. Thus the classiﬁcation predicate deﬁned in Section 3.1 can be used for this purpose. First, the closest point p ∈ χ0 to each sample point p ∈ χd is computed. This deﬁnes an empty sphere sp around p with radius p − q. If the point p only moves within this sphere during deformation, it is guaranteed not to intersect with the zero-region (see Figure 3.13). So additionally to exploiting spatial coherence as for boolean classiﬁcation, this approach also exploits the temporal coherence induced by the smooth deformation ﬁeld. The classiﬁcation predicate Ω has to be re-evaluated only when p leaves sp , which at the same time provides a new estimate for the updated sphere sp . Collision Handling There are diﬀerent ways to respond to a detected collision. The simplest solution is to undo the last deformation step and recover the surface geometry prior to the collision. Alternatively, the penetrating parts of the surface can be joined using a boolean union operation to maintain the validity of the surface. Figure 3.14 shows an editing session, where a deformation causes a self-intersection. After performing a boolean union, a sharp intersection curve is created as shown in (d). In the context of free-form deformation it is often more desirable to create a smooth transition between the two combined surface parts. Thus the particle simulation described in Section 3.1.4 can be used to blend the intersection region. CHAPTER 3. SHAPE MODELING 32 χ0 s1 p1 s2 χ0 χd p2 Figure 3.13: Temporal coherence for collision detection during deformation. The points p1 and p2 can move with the spheres s1 and s2 , resp., without intersecting the zero-region. (a) (d) (b) (e) (c) (f) Figure 3.14: Interactive modeling session with collision detection. (a - b) intermediate steps of the deformation, (c) collision detection, where the blue part has been detected as self-intersecting, (d), boolean union with sharp intersection curve, (e - f), particle-based blending with diﬀerent fall-oﬀ functions. 3.2.2 Dynamic Sampling Large deformations may cause strong distortions in the distribution of sample points on the surface that can lead to an insuﬃcient local sampling density. To prevent the point cloud from ripping apart and maintain a high surface quality, new samples have to be included where the sampling density 3.2. FREE-FORM DEFORMATION 33 becomes too low. This requires a method for measuring the surface stretch to detect regions of insuﬃcient sampling density. Then new sample points have to be inserted and their position on the surface determined. Additionally, scalar attributes, e.g., color values or texture coordinates, have to be preserved or interpolated. Measuring Surface Stretch The ﬁrst fundamental form known from diﬀerential geometry [dC76] can be used to measure the local distortion of a surface under deformation. Let u and v be two orthogonal tangent vectors of unit length at a sample point p. The ﬁrst fundamental form at p is deﬁned by the 2 × 2 matrix u·v u2 (3.3) u · v v2 The eigenvalues of this matrix yield the minimum and maximum stretch factors and the corresponding eigenvectors deﬁne the principal directions into which this stretching occurs. When applying a deformation, the point p is shifted to a new position p and the two tangent vectors are mapped to new vectors u and v . Local stretching implies that u and v might no longer be orthogonal to each other nor do they preserve their unit length. The amount of this distortion can be measured by taking the ratio of the two eigenvalues of Equation 3.3 (local anisotropy) or by taking their product (local change of surface area). When the local distortion becomes too strong, new samples have to be inserted to re-establish the prescribed sampling density. Since Equation 3.3 deﬁnes an ellipse in the tangent plane centered at p with the principal axes deﬁned by the eigenvectors and eigenvalues, p can be replaced by two new samples p1 and p2 that are positioned on the main axis of the ellipse (see Figure 3.15). (a) (b) Deformation Splitting (c) Relaxation Figure 3.15: Dynamic sampling. Top row: Deformation of a plane. (a) local stretching: blue corresponds to zero stretch, while red indicates maximum stretch, (b) surface after re-sampling, (c) sampling distribution. Bottom row: illustration of point insertion. CHAPTER 3. SHAPE MODELING 34 3.2.3 Filter Operations Whenever a splitting operation is applied, both the geometric position and the scalar function values for the newly generated sample points have to be determined. Both these operations can be described as the application of a ﬁltering operator: A relaxation ﬁlter determines the sample positions while an interpolation ﬁlter is applied to obtain the function values. Relaxation Introducing new sample points through a splitting operation creates local imbalances in the sampling distribution. To obtain a more uniform sampling pattern, a relaxation operator is applied that moves the sample points within the surface (see Figure 3.15). Similar to [Tur92] (see also Section 2.3) a simple point repulsion scheme is used with a repulsion force that drops linearly with distance. This conﬁnes the radius of inﬂuence of each sample point to its local neighborhood, which allows very eﬃcient computation of the relaxation forces. The resulting displacement vector is then projected into the points tangent plane to keep the samples on the surface. Interpolation Once the position of a new sample point p is ﬁxed using the relaxation ﬁlter, the associated function values need to be determined. This can be achieved using an interpolation ﬁlter by computing a local average of the function values of neighboring sample points. The relaxation ﬁlter potentially moves all points of the neighborhood of p. This tangential drift leads to distortions in the associated scalar functions. To deal with this problem a copy of each point that carries scalar attributes is created and its position is ﬁxed during relaxation. In particular, for each sample that is split a copy is maintained with its original data. These points will only be used for interpolating scalar values, they are not part of the current geometry description. Since these samples are dead but their function values still live, they are called zombies. Zombies will undergo the same transformation during a deformation operation as living points, but their positions will not be altered during relaxation. Thus zombies accurately describe the scalar attributes without distortions. Therefore, zombies are only used for interpolation, while for relaxation only living points are considered. After an editing operation is completed, all zombies will be deleted from the representation. Figure 3.16 illustrates this dynamic re-sampling method for a very large deformation that leads to a substantial increase in the number of sample points. While the initial plane consists of 40,000 points, the ﬁnal model contains 432,812 points, clearly demonstrating the robustness and scalability of the method in regions of extreme surface stretch. 3.2.4 Down-Sampling Apart from lower sampling density caused by surface stretching, deformations can also lead to an increase in sampling density, where the surface is squeezed. It might be desirable to eliminate samples in such regions while editing, to keep the overall sampling distribution uniform. However, dynamically removing samples also has some drawbacks. Consider a surface that is ﬁrst squeezed and then stretched back to its original shape. If samples get removed during squeezing, surface information such as color will be lost, which leads to increased blurring when the surface is re-stretched. Thus instead of dynamic sample deletion, an optional garbage collection is performed at the end of the editing operation. To reduce the sampling density, any of the simpliﬁcation methods of Chapter 2 can be used. 3.2. FREE-FORM DEFORMATION Deformation Splitting 35 Relaxation Interpolation Deletion of zombies New Samples Zombie Figure 3.16: A very large deformation using a combination of translatory and rotational motion. The left column shows intermediate steps with the top image indicating zero- and one-regions. Each point of the surface carries texture coordinates, which are interpolated during re-sampling and used for texturing the surface with a checkerboard pattern. The bottom row illustrates this interpolation process, where the function values are indicated by vertical lines. 36 CHAPTER 3. SHAPE MODELING Chapter 4 Appearance Modeling Mark Pauly The previous chapter has discussed boolean operations and free-form deformation for point-based shape modeling. Apart from the mere geometric shape, which is described by the position of the sample points in space, 3D objects also carry a number of additional attributes, such as color or material properties, that determine the overall appearance of the surface. In this chapter, a number of tools and algorithms for interactively editing surface appearance attributes will be discussed [ZPKG02]. These methods can be understood as a generalization of common photo editing techniques from 2D images to 3D point-sampled surfaces. 4.1 Overview This section gives an overview of the appearance modeling functionality for point-sampled models implemented in Pointshop3D (www.pointshop3d.com) by deﬁning a surface editing operation on an abstract level. A brief analysis of 2D photo editing identiﬁes three fundamental building blocks of an interactive editing operation: Parameterization, re-sampling, and editing. It will be shown that these concepts can be extended from 2D photo editing to 3D surface editing. For this purpose an operator notation will be introduced that allows a wide variety of editing operations to be deﬁned in a uniﬁed and concise way. The fundamental diﬀerences between functional images and manifold surfaces lead to diﬀerent implementations of these operators, however. 4.1.1 2D Photo Editing A 2D image I can be considered as a discrete sample of a continuous image function containing image attributes such as color or transparency. Implicitly, the discrete image always represents a continuous image, yet image editing operations are typically performed directly on the discrete samples. A general image editing operation can be described as a function of a given image I and a brush image B. The brush image is used as a generic tool to modify the image I. Depending on the considered operation it may be interpreted as a paint brush or a discrete ﬁlter, for example. The editing operation involves the following steps: First, a parameter mapping Φ has to be found that aligns the image I with the brush B. For example, Φ can be deﬁned as the translation that maps the pixel at the current mouse position to the center of B. Next, a common sampling grid for I and B has to be established, such that there is a one-to-one correspondence between the discrete samples. This requires a re-sampling operator Ψ that ﬁrst reconstructs the continuous image function and then samples this function on the common grid. Finally, the editing operator Ω combines the image samples with the brush samples using the one-to-one correspondence established before. The 37 CHAPTER 4. APPEARANCE MODELING 38 resulting discrete image I is then obtained using a concatenation of the operators described above as I = Ω(Ψ(Φ(I)), Ψ(B)). The goal is to generalize this operator framework to irregular point-sampled surfaces, as illustrated in Figure 4.1. Formally, this can be done by replacing the discrete image I by a point cloud P that represents a surface S. The discrete sample points can be considered as a direct extension of image pixels, carrying the attributes shown in Figure 4.4 (a). This motivates the term surfel, short for surface element, similar to pixel, which stands for picture element (see also [ZPvG01]). The transition from image to surface has the following eﬀects on the individual terms of the operator notation introduced above: Φ Parameterized patch Φ(S) Ψ Resampled patch Ψ(Φ(S)) Original point-based surface Brush Ψ(B) Ω Modified patch Modified point-based surface Figure 4.1: Overview of the operator framework for point-based surface editing. Parameterization Φ For photo editing, the parameter mapping Φ is usually speciﬁed by a simple, global 2D to 2D aﬃne mapping, i.e., a combination of translation, scaling, and rotation. Mapping a manifold surface onto a 2D domain is much more involved, however. Therefore, the user interactively selects subsets, or patches, of S that are parameterized individually. In general, such a mapping leads to distortions that cannot be avoided completely. Section 4.2 will describe two algorithms for computing a parameterization that correspond to two diﬀerent interaction schemes: Parameterization by orthogonal projection for interactive brush painting, and a method to compute a constrained minimum distortion parameterization. The latter allows the user to control the mapping in an intuitive manner by setting corresponding 4.1. OVERVIEW 39 feature points both on the parameter plane and the surface, respectively. Re-sampling Ψ Images are usually sampled on a regular grid, hence signal processing methods can be directly applied for re-sampling. However, the sampling distribution of surfaces is in general irregular, requiring alternative methods for reconstruction and sampling. For this purpose a parameterized scattered data approximation can be used that reconstructs a continuous function from the samples (see [ZPKG02]). This continuous function can then be evaluated at the desired sampling positions. Editing Ω Once the parameterization is established and re-sampling has been performed, all computations take place on the discrete samples in the 2D parameter domain. Hence the full functionality of photo editing systems can be applied for texturing and texture ﬁltering. Additionally, operations that modify the geometry, e.g., sculpting or geometry ﬁltering, can easily be incorporated into the system. As will be described in Section 4.1.3, all of these tools are based on the same simple interface that speciﬁes an editing tool by a set of bitmaps and few additional parameters. For example, a sculpting tool is deﬁned by a 2D displacement map, an alpha mask and an intrusion depth. 4.1.2 Interaction Modes The appearance attributes of a point-sampled model can be manipulated using two diﬀerent interaction schemes: Brush Interaction In this interaction mode the user moves a brush device over the surface and continuously triggers editing events, e.g., painting operations (see Figure 4.2). The brush is positioned using the mouse cursor and aligned with the surface normal at the current interaction point. This means that the parameterization is continuously and automatically re-computed and re-sampling is performed for each editing event. A complete editing operation is then performed using a ﬁxed brush image. orthogonal projection sample points parameter plane Figure 4.2: Brush interaction: Left: Cursor movement, middle: Parameterization by orthogonal projection onto the brush plane (2D for illustration), right: Painted surface. Selection Interaction Here the user ﬁrst selects a subset of the surface called a patch and deﬁnes the parameter mapping interactively by imposing point constraints (see Figure 4.3). Based on this ﬁxed parameterization, CHAPTER 4. APPEARANCE MODELING 40 various editing operations can be applied. Hence parameterization and re-sampling operators are evaluated once, while diﬀerent editing operators can be applied successively. Figure 4.3: Selection Interaction: Left: Feature points on parameter plane, middle: Feature points on surface, right: Final texture-mapped surface. 4.1.3 Brush Interface All appearance modeling operations are based on a generic brush interface (see also Figure 4.1). A brush is deﬁned as a M × N grid B, where each grid point bmn stores all the surface appearance attributes shown in Figure 4.4 (a). Each individual bitmap, e.g., the diﬀuse color image {cmn }, deﬁnes a brush channel that represents a corresponding continuous attribute function, similar to the surface samples P that represent the continuous surface S. Additionally, each brush channel carries an alpha mask that can be used for blending the brush coeﬃcients with the surface coeﬃcients as described in Section 4.4. The channel {dmn } deﬁnes a bitmap of displacement coeﬃcients that can be used for sculpting operations, such as normal displacement or carving. Figure 4.4 shows a typical brush that combines texture, geometry and material properties to support ﬂexible editing operations. 4.2 Parameterization This section describes two diﬀerent methods to compute a parameterization for a point-sampled surface that correspond to the two interaction schemes deﬁned above. For brush interaction the parameter mapping will be computed by a simple orthogonal projection, while an optimization method is applied for computing a constrained minimum distortion parameterization for selection interactions (see also [ZPKG02]). To deﬁne the parameterization Φ, the user ﬁrst selects a subset S of the surface S, described by a point cloud P ⊆ P . A mapping Φ : P → [0, 1] × [0, 1] is then computed that assigns parameter coordinates ui to each point pi ∈ P . 4.2.1 Orthogonal Projection A simple method for computing a parameter mapping is dimension reduction. 2D parameter values for a point pi ∈ P can be obtained by simply discarding one of the three spatial coordinates. With an appropriate prior aﬃne transformation, this amounts to an orthogonal projection of the sample points onto a plane. This plane can either be speciﬁed by the user, or computed automatically according to the distribution of the sample points, e.g., as a least-squares ﬁt. Using covariance analysis (see Section 1.2), the normal vector of the parameter plane would then be chosen as the eigenvector of the covariance matrix with smallest corresponding eigenvalue. Figure 4.5 shows examples of texturemapping operations using a parameterization obtained by orthogonal projection. In general, such a mapping will exhibit strong distortions and discontinuities, leading to inferior editing results. However, 4.2. PARAMETERIZATION 41 t en lor p o ec s iffu em lac al m or dis ec d n sp { bmn } = { cmn, dmn, smn, ula r olo rc t t t ien ien en ffic ffic ici f e e f o o e rc tc co ss ne se ula ien c u b f e ini f i m h p a d s s ka,mn, kd,mn, ks,mn, smn} (a) (b) Figure 4.4: Brush interface: (a) a brush speciﬁed by a set of bitmaps, (b) the brush applied to a surface. The zooms on the right show the surface under diﬀerent illumination to illustrate how the reﬂectance properties have been modiﬁed. if the surface patch is suﬃciently small, distortions will be small too and no discontinuities will occur. Thus orthogonal projection is a suitable parameterization method for brush interactions, where the parameter plane is deﬁned by the surface normal at the tool cursor and the surface patch is deﬁned by the projection of the brush onto the surface, as shown in Figure 4.2. (a) (b) Figure 4.5: The Max Planck model parameterized by orthogonal projection from the front. (a) texture mapped surface, (b) the color-coded ﬁrst derivative of the parameterization measures the stretch of the mapping, where blue corresponds to minimum, red to maximum stretch. 4.2.2 Constrained Minimum Distortion Parameterization As Figure 4.5 illustrates, orthogonal projection leads to strong distortions in the parameterization. Furthermore, it provides little support for interactive control of the mapping by the user. Consider the CHAPTER 4. APPEARANCE MODELING 42 typical texture mapping operation shown in Figure 4.3. The goal is to map a 2D image of a human face onto a laser-range scan of a diﬀerent face. It is certainly desirable to minimize the distortion of the mapping. Equally important, however, is a good correspondence of feature points, e.g., the tip of the nose in the image should be mapped onto the tip of the nose on the surface. Thus some mechanism is needed that allows the user to deﬁne corresponding feature points both in the image and on the surface. These point-to-point correspondences are then incorporated as constraints into an optimization that computes the mapping [Lev01]. First, an objective function for a continuous surface patch is deﬁned that penalizes high distortion as well as the approximation error of the feature point constraints. A suitable discretization then yields a system of linear equations that can be solved using conjugate gradient methods. Objective Function A continuous parameterized surface SX can be deﬁned by a mapping X : [0, 1] × [0, 1] → SX ⊂ IR3 , which for each parameter value u = (u, v) ∈ [0, 1] × [0, 1] determines a point x = X(u) ∈ SX on the surface SX . The mapping X deﬁnes a parameterization of the surface SX . Let U = X −1 be the inverse mapping, i.e., a function that assigns parameter coordinates u to each point x ∈ SX . The distortion of the parameter mapping can be measured using the cost function γ(u)du, Cdist (X) = H where H = [0, 1] × [0, 1], γ(u) = θ and 2 ∂2 Xu (θ, r) dθ, ∂r2 cos θ . Xu (θ, r) = X u + r sin θ γ(u) is deﬁned as the integral of the squared second derivative of the parameterization in each radial direction at a parameter value u using a local polar re-parameterization Xu (θ, r). If γ(u) vanishes, the parameterization is arc length preserving, i.e., deﬁnes a polar geodesic map at u. Additionally, a set M of point-to-point correspondences can be speciﬁed such that a point pj of the point cloud corresponds to a point uj in the parameter domain. These point pairs serve as constraints that are approximated in a least-squares sense using the cost function Cf it (X) = (X(uj ) − pj )2 . j The two cost functions Cdist and Cf it can be combined into the objective function C(X) = Cf it (X) + β · Cdist (X), where β is an additional parameter that allows to control the relative weight of the ﬁtting error and distortion measure. This derivation of the objective function follows Levy’s method for triangle meshes [Lev01]. By replacing the mesh connectivity with a point neighborhood relation as deﬁned in Section 1.1, a discrete formulation of the objective function can be derived for point-sampled surfaces. This requires a discretization of the directional derivatives in γ(u), which can be obtained using divided diﬀerences on a discrete set of normal sections. For a complete derivation, the reader is referred to [ZPKG02]. When substituting X for U , the discrete objective function ﬁnally has the form 2 C̃(U ) = aj,i ui = b − Au2 , (4.1) bj − j i 4.2. PARAMETERIZATION 43 where u is the vector of all unknowns ui = (ui , vi )T . The coeﬃcients aj,i result from the discretization of the second derivatives and the bj are derived from the ﬁtting constraints. Using normal equations, the linear least squares problem of Equation 4.1 can be transformed to a sparse system of linear equations, which can be solved with conjugate gradient methods. Nested Iteration A standard approach for improving the convergence of iterative conjugate gradient solvers is nested iteration. The system is ﬁrst solved on a coarse representation and this solution is propagated to the ﬁner representation. Since each unknown in Equation 4.1 corresponds to a sample of the model point cloud, a coarser representation can be obtained using the clustering methods described in Section 2.1. This scheme can be recursively extended to a hierarchy of nested approximations as illustrated in Figure 4.6. Special care has to be taken to deﬁne the constraints on the coarser levels, as the original point pair correspondences were deﬁned on the ﬁnest level. A simple solution is to just propagate the constraint to the centroid of the cluster it belongs to. If a point carries more than one constraint at a certain level, all but one of the constraints are removed on this level to maintain the injectivity of the mapping. simplify by clustering simplify by clustering simplify by clustering set up linear system set up linear system set up linear system set up linear system solve equations solve equations solve equations solve equations Figure 4.6: Hierarchy used for nested iteration. The top-row shows the clusters color-coded on the original point cloud. CHAPTER 4. APPEARANCE MODELING 44 Discussion Figure 4.7 shows the inﬂuence of the parameter β in the objective function C(X). As the images illustrate, it allows the user to deﬁne a trade-oﬀ between the approximation of the ﬁtting constraints and the smoothness of the mapping. Floater has presented a diﬀerent approach to compute a parameterization for point-sampled surfaces using shape preserving weights [FR01]. In his method the system of equations is obtained by expressing each unknown as a convex combination of its neighbors. This means that the boundary of the surface has to be mapped onto a convex region in parameter space. In the above formulation no such constraints have been imposed, which allows the method to be used as an extrapolator. Note that at least three point-pair correspondences are required to deﬁne the mapping. Figure 4.8 shows two examples of a texture mapping operation using the minimum distortion Figure 4.7: Inﬂuence of the weighting parameter β of the objective function. The top images illustrate the feature point correspondences, the bottom row show the resulting mapping for diﬀerent values of β. parameterization. As described in Section 4.1.2, this type of operation requires the user to interactively deﬁne corresponding feature points both on the 2D image and on the 3D point-sampled surface, as shown on the left. Figure 4.9 illustrates the distortion of these parameterizations by mapping a regular square grid texture onto the surface. 4.3 Re-Sampling The mapping function Φ deﬁnes a parameter value ui for each pi ∈ P of the selected surface patch S . To create a one-to-one correspondence between the surface samples pi ∈ P and the brush samples bmn ∈ B, either the surface or the brush needs to be re-sampled. Note that the brush samples are implicitly parameterized, i.e., each bmn has parameter coordinates (m/M, n/N ) ∈ [0, 1]. 4.3.1 Re-sampling the Brush Re-sampling the brush is performed by evaluating the continuous brush function represented by the discrete brush bitmaps at the parameter values of the sample points of the surface patch. The most 4.3. RE-SAMPLING 45 Figure 4.8: Texture mapping using the minimum distortion parameterization. The images on the left show the corresponding feature points. On the right, the ﬁnal textured surface is shown. Figure 4.9: Illustration of local stretching for the texture mapping examples of Figure 4.8. The image on the right shows the color-coded ﬁrst derivative of the parameterization, where blue denotes minimum absolute value and red denotes maximum absolute value. simple reconstruction uses piecewise constant basis functions which amounts to nearest neighbor sampling. As Figure 4.10 (a) illustrates, this can lead to severe aliasing artifacts, which can be reduced by applying a Gaussian low-pass ﬁlter to the brush function prior to sampling. This ﬁltering is analogous CHAPTER 4. APPEARANCE MODELING 46 to the elliptical weighted average (EWA) ﬁltering used in point rendering and the reader is referred to [ZPvG01] for further details. Note that if the model surface is sampled substantially less densely than the brush, Gaussian ﬁltering will lead to signiﬁcant blurring. If the brush contains high-frequency detail, this information cannot be accurately mapped onto the surface without introducing new samples into the model. Figure 4.10: Re-sampling the brush on a model consisting of 40,880 points. (a) nearest-neighbor sampling, (b) - (d) Gaussian ﬁltering with increasing ﬁlter width. 4.3.2 Re-sampling the Surface To overcome the problem of information loss when re-sampling the brush, an alternative sampling method re-samples the surface to exactly map the M × N sampling grid of the brush. For this purpose a parameterized scattered data approximation is used. This method reconstructs a continuous surface from the discrete sample points, which can then be sampled at the parameter values of the brush samples. The idea is to compute local polynomial ﬁtting functions at each pi ∈ P that are blended using a mapping from the local parameter plane of the ﬁtting functions into the global parameter space of the surface (see [ZPKG02] for details). Figure 4.11 shows various examples of editing operations where the surface has been re-sampled according to the sampling grid of the brush. If the sampling resolution of the brush decreases, surface detail is lost. This is complementary to the situation described above, where brush information was lost due to insuﬃcient sampling density of the model surface. Figure 4.11: Re-sampling the surface. The sampling resolution of the brush varies from 500 × 500 pixels in (a), 250 × 250 pixels in (b), 100 × 100 pixels in (c), 50 × 50 to pixels in (d). 4.4 Surface Editing The re-sampling method of Section 4.3 provides samples of the surface SΨ = Ψ(Φ(S)) and of the brush BΨ = Ψ(B) with identical sampling distribution. Thus the two can be combined by applying an editing operator directly on the discrete coeﬃcients. Note that both SΨ and BΨ represents all the 4.4. SURFACE EDITING 47 sample attributes shown in Figure 4.4 (a). Depending on the intended functionality, an editing operator will then manipulate a subset of these surface attributes, such as diﬀuse color or spectral coeﬃcient. In the following some of the editing operators will be described that have been implemented in the Pointshop3D system. A prime will denote the manipulated attributes, e.g., xi describes the position of a sample of the edited surface. Quantities that stem from the brush are marked with a bar, e.g., c̄i is the diﬀuse color of a brush sample. All other variables are part of the surface function. 4.4.1 Painting Painting operations modify surface attributes by alpha-blending corresponding surface and brush coeﬃcients. For example, the diﬀuse color can be altered by applying the painting operator on the color values, i.e., ci = ᾱi · ci + (1 − ᾱi ) · c̄i , where ᾱi is an alpha value speciﬁed in the brush function (see Figure 4.12 (a)). Similarly, painting can be applied to other attributes. (a) (b) Figure 4.12: Editing operations on the Chameleon (101,685 points): (a) texture painting, where the ﬂowers shown on the left have been alpha-blended onto the surface, (b) texture ﬁltering, where an oil-paint ﬁlter has been applied to the left half of the model. 4.4.2 Sculpting The system supports two variations of sculpting operations that modify the geometry of the surface. The ﬁrst option is to apply normal displacements to the sample positions, i.e., xi = xi + d¯i · ni , where d¯i is a displacement coeﬃcient given in the brush function. As illustrated in Figure 4.14 (a), this type of editing operation is particularly suitable for embossing or engraving. On the other hand, a carving operation is motivated by the way artists work when sculpting with clay or stone. It implements a ”chisel stroke” that removes parts of the surface in the fashion of a boolean intersection (see also Section 3.1). The editing tool is deﬁned using a least squares plane ﬁtted at the touching point of the tool cursor and an intrusion depth. The new sample position is then given by xi = b̄i + d¯i · n̄ xi − b̄i < d¯i xi otherwise, where b̄i is the base point on the reference plane and n̄ is the plane normal (see Figure 4.13). Carving operations can also be applied to rough surfaces (see Figure 4.14 (b)), where normal displacements fail due to the strong variations of the surface normals. CHAPTER 4. APPEARANCE MODELING 48 4.4.3 Filtering Filtering is a special kind of editing operation that modiﬁes the samples of the original model using f a user-speciﬁed ﬁlter function f . First, the ﬁlter function is applied to SΨ yielding SΨ = f (SΨ ). Then ﬁltered and original attributes are combined using the brush function for alpha blending. As an example, consider texture ﬁltering, i.e., ci = ᾱ · cfi + (1 − ᾱ) · ci , where cfi is the ﬁltered color value (illustrated in Figure 4.12 (b)). The ﬁlter function is usually implemented as a discrete convolution. Therefore, arbitrary discrete linear ﬁlters can be implemented by simply choosing the appropriate kernel grid. Filters can be applied to any attribute associated with a sample, e.g., color, normal, or distance from the reference plane for geometric oﬀset ﬁltering. Note that ﬁltering with large kernels can be implemented eﬃciently in the spectral domain, similar to [PG01]. _ n original samples new samples _ bi original samples new samples Figure 4.13: Normal displacements vs. carving. (a) brush image, (b) normal displacement, (c) carving. Figure 4.14: Editing operations. Normal displacements (left) and carving on a rough surface (right). Chapter 5 Pointshop3D Mark Pauly This chapter describes Pointshop3D, a software platform for point-based shape and appearance modeling [ZPKG02, PKKG03]. Most of the algorithms presented in this course have been implemented and analyzed within this framework. A brief description of the main components is followed by a more detailed discussion on data structures for spatial queries. 5.1 System Overview Pointshop3D is designed as a modular software platform and implementation test-bed for point-based graphics applications. It provides a set of kernel functions for loading, storing, modifying, and rendering point-sampled surfaces. Most of the functionality is implemented in various tools and plug-ins that are brieﬂy discussed below. A more detailed description of the software architecture and user interface can be found in the online documentation available at www.pointshop3d.com. 5.1.1 Tools and Plug-ins The interaction metaphor of Pointshop3D is similar to common photo editing tools. Figure 5.1 shows a screen shot of the main user interface, where the most important interaction tools are annotated: • The color picker tool allows to extract sample attributes from the model. • The selection tool is used to select parts of the surface, e.g., to create a patch for selection interactions. • The navigation tool controls the camera position and the relative orientation of diﬀerent objects with respect to the world coordinate system. • The brush tool implements brush interactions. • The deformation tool supports free-form deformation as described in Section 3.2. • The eraser tool allows to remove parts of the surface. • The ﬁlter tool implements various ﬁlters (see Section 4.4.3). • The lighting tool controls the position of the light source. • The parameterization tool allows the speciﬁcation of point constraints for computing a minimum distortion parameterization (see Section 4.2.2). 49 CHAPTER 5. POINTSHOP3D 50 • The template tool is a place-holder that supports easy extensions of the system by adding new tools. • The What’s this? tool provides an online help. • The color chooser tool allows to select an rgb color from a palette for brush painting. • The brush builder is used to create new brushes (see Figure 4.4) for brush interactions. • The brush chooser stores a list of pre- or user-deﬁned brushes used for brush interactions. • The tool settings viewer displays additional parameters of the currently active tool. • The preview renderer icon activates the preview renderer. • The object chooser selects a certain object within the scene. Brush Builder Brush Chooser Tool Settings Viewer Color Choser Tool What's this? Tool Preview Renderer Object Chooser Color Picker Tool Selection Tool Navigation Tool Brush Tool Deformation Tool Eraser Tool Filter Tool Lighting Tool Parameterization Tool Template Tool Figure 5.1: The main Pointshop3D interface. Each of these tools can be conﬁgured as described in the online documentation of Pointshop3d. Additional functionality, e.g., a watermarking method, is implemented as plug-ins that are accessible via the main menu. Tools and plug-ins are implemented as dynamic link libraries and can be loaded dynamically at run-time, which makes the system easily extensible. 5.2. DATA STRUCTURES 5.2 51 Data Structures The central computational primitive required by the algorithms described in the previous chapters is closest points query: Given a point x ∈ IR3 , ﬁnd the point(s) pi ∈ P , such that x − pi is minimal. Closest point queries occur in two diﬀerent contexts: • The neighborhood relation deﬁned in Section 1.1 is based on the k-closest points of a sample in P . In this case the query point is part of the set of point samples, i.e., x ∈ P . The number of required closest points ranges from 8 to 20 for, e.g., normal estimation (Section 1.2), to up to about 200 for multi-scale variation estimation for feature classiﬁcation [PKG03]. • Spatial queries are also required for the MLS projection (Section 1.3), boolean classiﬁcation (Section 3.1), and collision detection (Section 3.2.1). Here the query point can be arbitrary, i.e., in general, and typically only a single closest point is required. The performance of the algorithms implemented in Pointshop3D critically depends on eﬃcient data structures for such queries. The requirements for these data structures vary considerably, depending on the speciﬁc context. For example, multi-scale feature classiﬁcation [PKG03] applies nearest neighbor queries with varying neighborhood size for static objects, while particle simulation (Section 2.3) requires closest point queries for completely dynamic point distributions. Other operations, e.g., the ﬁltering methods applied during free-form deformation (Section 3.2) operate on models with dynamic point locations, but mostly constant neighborhood relations. In the design of appropriate data structures, one is typically faced with trade-oﬀ between eﬃcient construction/updating of the data structure and fast query response. In Pointshop3D two main data structures, kd-trees and dynamic grids, have been implemented. Kd-trees feature fast query response, yet perform poorly when new points need to be inserted or existing points updated or removed. Dynamic grids provide eﬃcient updating at the expense of slower query response times. 5.2.1 Kd-Trees Bentley introduced kd-trees as a generalization of binary search trees in higher dimensions [Ben75]. A number of improvements have been proposed since then, see for example [FBF77, Spr91]. Kd-trees are binary trees, where each node is associated with a cubical cell and a plane orthogonal to one of the three coordinate axes. The plane splits the cell into two sub-cells, which correspond to two child nodes in the tree. The cells of the leaf nodes are called buckets and represent the whole space. In Pointshop3D, kd-trees are built in a recursive top-down approach, where the sliding-midpoint rule is used to determine the split axis (see [AF00]). Average time complexity for the construction of the tree is O(n log n) for a point cloud consisting of n sample points. To compute the k-closest points from a given location x ∈ IR3 , the tree is traversed from the root until the bucket containing x is found. From this leaf node, the k-closest points are determined by back-trapping towards the root. Finding the k-closest points takes O(k + log n) time on average, insertions and updates of single points require time O(log n) [Sam90]. 5.2.2 Dynamic Grids Dynamic grids have been implemented as a data structure for closest point queries in dynamic settings. The idea is to subdivide space into a regular grid of cubical cells, each of which stores a list of all points that fall within its cell [Hec97]. Range queries can be performed by scan-converting the query sphere to determine all cells that potentially contain samples located within the query range. Each of these cells is then processed by testing all points of the cell against the query sphere. To accommodate variations of point density over time, the grid can be re-structured dynamically as described in [Hec97]. Finding the correct grid cell for a point x ∈ IR3 that lies within the object’s bounding box takes O(1) time. Thus a dynamic grid for n samples can be constructed in O(n) time and the insertion and CHAPTER 5. POINTSHOP3D 52 Data Structure List Grid Kd-tree Construction O(n) O(n) O(n log n) Insertion O(1) O(1) O(log n) Update O(1) O(1) O(log n) Query O(n) O(m) O(log n) Table 5.1: Average time complexity for spatial data structures used for closest point queries. n is the number of points in P , m is the average number of points in each grid cell. Update refers to a change of sample position or a sample point deletion. updating of a single point takes O(1) time. Since each grid cell stores a linear list of points, the average time complexity of a query is O(m), where m is the average number of points per cell. Note that a linear list of all sample points is a special case of a grid data structure consisting of only one grid cell. Figure 5.2 shows a 2D illustration of the kd-tree and grid data structures. Table 5.1 compares the average time complexity of these data structures for typical operations used in the algorithms of Pointshop3D. 4 1 3 5 2 7 6 8 (a) (b) Figure 5.2: Spatial data structures for closest point queries, where the red dot indicates the query location and the black circle shows the query sphere. (a) kd-tree, where the numbers indicate the order in which the buckets are traversed, (b) dynamic grid, where the gray cells mark the intersection with the query sphere. 5.2.3 Neighborhood Caching For many processing methods the k-closest points of a sample point have to be determined to establish local neighborhood relations (see Section 1.1). Often the point cloud is static or semi-dynamic, i.e., point locations vary but neighborhood relations remain (mostly) constant. If neighborhood relations have to be used multiple times, the eﬃciency of the computations can be improved considerably by caching these neighborhoods. Each sample point explicitly stores a list of its k-nearest neighbors computed using a kd-tree or a dynamic grid. This deﬁnes an adjacency graph that is comparable to a connectivity graph for triangle meshes. Unlike the latter, the adjacency graph does not deﬁne a consistent manifold surface, however. Thus the construction of the adjacency graph is signiﬁcantly more eﬃcient than the reconstruction of a triangle mesh from a given point cloud. 5.2.4 Spatial Queries in Pointshop3D This section discusses the use of the data structures described above in the algorithms of Pointshop3D. 5.2. DATA STRUCTURES 53 Surface Simplification Spatial data structures for surface simpliﬁcation have already been discussed in Section 2.4.4. Incremental clustering (Section 2.1.1) uses kd-trees for nearest neighbor queries, since the point cloud is static. Hierarchical clustering (Section 2.1.2) builds a BSP-tree and does not require closest point queries. Iterative simpliﬁcation (Section 2.2) builds an explicit adjacency graph using kd-trees during initialization, which is maintained during simpliﬁcation. Particle simulation (Section 2.3) makes use of dynamic grids, since individual particles experience signiﬁcant drifts along the surface, which leads to dynamically changing point neighborhoods. Boolean Classification For boolean operations (Section 3.1), two static objects are positioned relative to each other by the user. Since the resulting aﬃne transformation can be directly incorporated into the query, no updates of point locations are required. Thus closest point queries for classiﬁcation are implemented using kd-trees. Free-Form Deformation For free-form deformation (Section 3.2) closest point queries are required to evaluate the distance functions dj (see Equation 3.2) for computing the scale factor t. This is done once at the beginning of the modeling session, so no dynamic updates are required while the user interactively deforms the surface. Thus a kd-tree is most suitable for scale factor computation. Collision Detection For collision detection (Section 3.2.1) the classiﬁcation predicate Ω (see Equation 3.1) has to be evaluated for dynamically varying point locations. Since performance is most critical, the algorithm only detects collisions of the deformable region with the zero-region. Since the latter is described by a static point cloud, kd-trees can be used for eﬃcient closest point queries. Thus the collision detection algorithm in its current version cannot detect collisions of the deformable region with itself. Particle-Based Blending The blending method based on oriented particles (Section 3.1.4) uses the dynamic grid data structure. Since the particle simulation is mostly used to smooth out the sharp creases created by boolean operations (see Figure 3.9), particles experience a small spatial displacement during the simulation. Thus the neighborhood relations typically remain constant for a certain number of iterations. This can be exploited to improve the performance by caching the neighborhoods as discussed above. Dynamic Re-sampling When dynamically sampling the surface during deformation (Section 3.2.2), the relaxation and interpolation ﬁlters require nearest neighbor information for samples in the deformable region. Since these samples vary in position and new samples are inserted dynamically, kd-trees cannot be used eﬃciently. The dynamic grid data structure has also been found to be unsuitable, due to the relatively high cost of a spatial query. The relaxation ﬁlter shifts the point locations only slightly, which typically does not change the neighborhood relation signiﬁcantly. Therefore, neighborhoods are computed at the beginning of the interaction and cached during deformation (see above). When new points are inserted, neighborhoods have to be updated, however. Since new samples are created by splitting an existing sample into two, the corresponding neighborhood relations can be propagated from the parent sample to its child samples. To maintain neighborhood consistency, each sample stores bidirectional neighborhood relations, i.e., a list of its neighbors and a list of samples of which itself is a neighbor. 54 CHAPTER 5. POINTSHOP3D Bibliography [AA03a] Adamson A., Alexa M.: Approximating and intersecting surfaces from points. In Proceedings of the Eurographics/ACM SIGGRAPH symposium on Geometry processing (2003), Eurographics Association, pp. 230–239. [AA03b] Adamson A., Alexa M.: Ray tracing point set surfaces. In Proceedings of Shape Modeling International (2003). [ABCO∗ 01] Alexa M., Behr J., Cohen-Or D., Fleishman S., Levin D., Silva C. T.: Point set surfaces. In Proceedings of the conference on Visualization ’01 (2001). [ABK98] Amenta N., Bern M., Kamvysselis M.: A new Voronoi-based surface reconstruction algorithm. Computer Graphics 32, Annual Conference Series (1998), 415–421. [AF00] Arya S., Fu H.-Y. A.: Expected-case complexity of approximate nearest neighbor searching. In Proceedings of the eleventh annual ACM-SIAM symposium on Discrete algorithms (2000), Society for Industrial and Applied Mathematics, pp. 379–388. [AS94] Agarwal P. K., Suri S.: Surface approximation and geometric partitions. In Proceedings of the ﬁfth annual ACM-SIAM symposium on Discrete algorithms (1994), Society for Industrial and Applied Mathematics, pp. 24–33. [Bar84] Barr A. H.: Global and local deformations of solid primitives. In Proceedings of the 11th annual conference on Computer graphics and interactive techniques (1984), ACM Press, pp. 21–30. [Ben75] Bentley J. L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18, 9 (1975), 509–517. [BW00] Brodsky D., Watson B.: Model simpliﬁcation through reﬁnement. In Graphics Interface (May 2000), pp. 221–228. [CRS] Cignoni P., Rocchini C., Scopigno R.: Metro: Measruring error on simpliﬁfed surfaces. In Computer Graphics Forum, pp. 167–174. [dC76] do Carmo M. P.: Diﬀerential Geometry of Curves and Surfaces. Prentice Hall, 1976. [FBF77] Freidman J. H., Bentley J. L., Finkel R. A.: An algorithm for ﬁnding best matches in logarithmic expected time. ACM Trans. Math. Softw. 3, 3 (1977), 209–226. [FCOAS03] Fleishman S., Cohen-Or D., Alexa M., Silva C. T.: Progressive point set surfaces. ACM Transactions on Graphics (TOG) 22, 4 (2003), 997–1011. [FR01] Floater M. S., Reimers M.: Meshless parameterization and surface reconstruction. Comput. Aided Geom. Des. 18, 2 (2001), 77–92. 55 BIBLIOGRAPHY 56 [Gar99] Garland M.: Quadric-Based Polygonal Surface Simpliﬁcation. PhD thesis, Computer Science Department, Carnegie Mellon University, 1999. [GH97] Garland M., Heckbert P. S.: Surface simpliﬁcation using quadric error metrics. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques (1997), ACM Press/Addison-Wesley Publishing Co., pp. 209–216. [GJ02] Giesen J., John M.: Surface reconstruction based on a dynamical system. Computer Graphics Forum 21 (2002). [HDD∗ 94] Hoppe H., DeRose T., Duchamp T., Halstead M., Jin H., McDonald J., Schweitzer J., Stuetzle W.: Piecewise smooth surface reconstruction. Computer Graphics 28, Annual Conference Series (1994), 295–302. [Hec97] Heckbert P. S.: Fast Surface Particle Repulsion. Tech. rep., CMU Computer Science, 1997. [Hop96] Hoppe H.: Progressive meshes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (1996), ACM Press, pp. 99–108. [Jol86] Jolliffe I.: Principal Component Analysis. Springer Verlag, 1986. [KCVS98] Kobbelt L., Campagna S., Vorsatz J., Seidel H.-P.: Interactive multi-resolution modeling on arbitrary meshes. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques (1998), ACM Press, pp. 105–114. [KM97] Krishnan S., Manocha D.: An eﬃcient surface intersection algorithm based on lowerdimensional formulation. ACM Trans. Graph. 16, 1 (1997), 74–106. [Lev98] Levin D.: The approximation power of moving least-squares. Math. Comput. 67, 224 (1998), 1517–1531. [Lev01] Levy B.: Constrained texture mapping for polygonal meshes. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (2001), ACM Press, pp. 417–424. [Lev03] Levin D.: Mesh-independent surface interpolation. Geometric Modeling for Scientiﬁc Visualization (2003). [Lin01] Linsen L.: Point Cloud Representation. Tech. rep., Faculty of Computer Science, University of Karlsruhe, 2001. [LPC∗ 00] Levoy M., Pulli K., Curless B., Rusinkiewicz S., Koller D., Pereira L., Ginzton M., Anderson S., Davis J., Ginsberg J., Shade J., Fulk D.: The digital michelangelo project: 3d scanning of large statues. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques (2000), ACM Press/AddisonWesley Publishing Co., pp. 131–144. [LS86] Lancaster P., Salkauskas K.: Curve and Surface Fitting: An Introduction. Academic Press, 1986. [PG01] Pauly M., Gross M.: Spectral processing of point-sampled geometry. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (2001), ACM Press, pp. 379–386. [PGK02] Pauly M., Gross M., Kobbelt L. P.: Eﬃcient simpliﬁcation of point-sampled surfaces. In Proceedings of the conference on Visualization ’02 (2002), pp. 163–170. BIBLIOGRAPHY 57 [PKG02] Pauly M., Kobbelt L., Gross M.: Multiresolution Modeling of Point-Sampled Geometry. Tech. rep., ETH Zurich, Department of Computer Science, 2002. [PKG03] Pauly M., Keiser R., Gross M.: Multi-scale feature extraction on point-sampled surfaces. Computer Graphics Forum 22 (2003), 281–289. [PKKG03] Pauly M., Keiser R., Kobbelt L. P., Gross M.: Shape modeling with point-sampled geometry. ACM Transactions on Graphics (TOG) 22, 3 (2003), 641–650. [RB93] Rossignac J., Borrel P.: Multi-resolution 3d approximations for rendering complex scenes. In Modeling in Computer Graphics (1993), pp. 455–465. [Sam90] Samet H.: The Design and Analysis of Spatial Data Structures. Addison-Wesley, 1990. [Sed98] Sedgewick R.: Algorithms in C++ (3rd edition). Addison Wesley, 1998. [SG01] Shaffer E., Garland M.: Eﬃcient adaptive simpliﬁcation of massive meshes. In Proceedings of the conference on Visualization ’01 (2001), IEEE Computer Society, pp. 127– 134. [SP86] Sederberg T. W., Parry S. R.: Free-form deformation of solid geometric models. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques (1986), ACM Press, pp. 151–160. [Spr91] Sproull R. L.: Reﬁnements of nearest-neighbour searching in k-dimensional trees. Algorithmica 6 (1991), 579–589. [ST92] Szeliski R., Tonnesen D.: Surface modeling with oriented particle systems. In Proceedings of the 19th annual conference on Computer graphics and interactive techniques (1992), ACM Press, pp. 185–194. [Tur92] Turk G.: Re-tiling polygonal surfaces. In Proceedings of the 19th annual conference on Computer graphics and interactive techniques (1992), ACM Press, pp. 55–64. [Tur01] Turk G.: Texture synthesis on surfaces. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (2001), ACM Press, pp. 347–354. [WH94] Witkin A. P., Heckbert P. S.: Using particles to sample and control implicit surfaces. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques (1994), ACM Press, pp. 269–277. [ZPKG02] Zwicker M., Pauly M., Knoll O., Gross M.: Pointshop 3d: an interactive system for point-based surface editing. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques (2002), ACM Press, pp. 322–329. [ZPvG01] Zwicker M., Pfister H., van Baar J., Gross M.: Surface splatting. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (2001), ACM Press, pp. 371–378. [ZRB∗ 04] Zwicker M., Rsnen J., Botsch M., Dachsbacher C., Pauly M.: accurate splatting. In Proceedings of Graphics Interface (2004). Perspective

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

### Related manuals

Download PDF

advertisement