School of Computing FACULTY OF ENGINEERING Tabletop Robotics: Board Games with a Baxter Robot Pascal Alexander Siddons Submitted in accordance with the requirements for the degree of BSc Artificial Intelligence Session 2014/2015 - ii - The candidate confirms that the following have been submitted: Items Format Recipient(s) and Date Final Report (2 copies) Report SSO (13/05/15) Final Report (digital) Report VLE (13/05/15) Demonstration video URL/File Download Prof. Tony Cohn & Dr Yiannis Gatsoulis (13/05/15) Type of Project: Exploratory Software The candidate confirms that the work submitted is their own and the appropriate credit has been given where reference has been made to the work of others. I understand that failure to attribute material which is obtained from another source may be considered as plagiarism. (Signature of student) ______________________________ © 2015 The University of Leeds and Pascal Alexander Siddons - iii - Summary The aim of this project is to write software for a Baxter robot in order to make it capable of playing a board game with a human player, playing fully legal and non-random moves with some element of strategy. This requires a suitable game to be designed and prototyped with components suitable to be handled by a Baxter robot. - iv - Acknowledgements I would like to thank my project supervisors, Professor Tony Cohn and Dr Yiannis Gatsoulis for all the advice, time and support they have given me throughout this project, as well as the other researchers working in the robotics lab for putting up with me while I was there. I would also like to thank my partner Paulina for keeping my sanity in check as the final deadline loomed and playtesting the game with me, as well as Ben Hirsh for letting me park outside his house, which is considerably closer to campus than mine. I would like to thank the late Sir Terry Pratchett for his literature and words of wisdom, which I grew up with and have instilled in me a great fascination with the wonderful and the out of the ordinary, the cutting-edge and the awe-inspiring, without which I’d probably be doing a project involving mid-range predictions in finance or something equally dreadful. The manner in which he wrote about death (or rather, DEATH) also made his passing away partway through this project much easier to cope with. Finally, I would like to give thanks to Tom McMurchie and Calliope Games, for designing and publishing Tsuro respectively, which heavily inspired the game that this project revolves around, and has given me hours of enjoyment. -v- Table of Contents Summary .............................................................................................................. iii Acknowledgements ............................................................................................. iv Table of Contents .................................................................................................. v Chapter 1 Introduction.......................................................................................... 1 1.1 Aims ...................................................................................................... 1 1.2 Objectives .............................................................................................. 1 1.3 Minimum Requirements ......................................................................... 1 1.4 The Problem .......................................................................................... 1 1.5 Methodology .......................................................................................... 3 1.6 Project Management ................................................................................ 3 Chapter 2 Background And Literature Review ................................................... 6 2.1 Game AI ................................................................................................... 6 2.1.1 Games From a Computing Perspective ........................................ 6 2.1.2 Adversarial Searches and Pruning ................................................ 6 2.1.2 Machine Learning ......................................................................... 7 2.2 Machine Vision ......................................................................................... 8 2.3 Robotics ................................................................................................... 8 2.3.1 Robotic Behaviour Architectures ................................................... 8 2.3.2 Inverse Kinematics........................................................................ 9 Chapter 3 Materials ............................................................................................. 11 3.1 The Robot .............................................................................................. 11 3.2 The Game .............................................................................................. 12 3.2.1 Adapting the game to work with Lucas ........................................ 13 3.2.2 Strategies ................................................................................... 13 3.3 Original Concept..................................................................................... 15 3.4 Dealing With Limitations ......................................................................... 15 3.4.1 Vision .......................................................................................... 15 3.4.2 Mechanical ................................................................................. 17 3.5 Consequences of Compromise on the Game ......................................... 18 3.6 Breaking Down the Game Logic ............................................................. 18 3.7 Decisions on Development Tools ........................................................... 21 Chapter 4 Methods .............................................................................................. 22 4.1 The Vision System.................................................................................. 22 - vi 4.1.1 Block Identification ...................................................................... 22 4.1.2 Scanning the Board State ........................................................... 25 4.2 The Mechanical System ......................................................................... 26 4.3 The Game Logic System ........................................................................ 28 4.4 Putting It All Together ............................................................................. 30 Chapter 5 Analysis and Evaluation of the System............................................ 33 5.1 System Reliability ................................................................................... 33 5.2 Effectiveness .......................................................................................... 34 5.2.1 Effectiveness of the Vision Systems ........................................... 35 5.2.2 Effectiveness of the Mechanical Systems ................................... 35 5.3 Limitations Imposed By the System ............................................... 35 5.4 Gameplay Data ...................................................................................... 36 5.5 Discussion of Results ............................................................................. 38 5.5.1 Data Gathering Methods ............................................................. 38 5.5.2 Data Analysis .............................................................................. 38 5.6 Evaluation of the Project ......................................................................... 38 5.6.1 Project Schedule ......................................................................... 38 5.6.2 Choice of Development Tools ..................................................... 39 5.6.3 Decisions in Game and Component Design ................................ 39 5.6.4 Decisions in Software Design ..................................................... 39 Chapter 6 Conclusions ....................................................................................... 40 6.1 Conclusions ............................................................................................ 40 6.2 Future Work............................................................................................ 42 List of References ............................................................................................... 44 Appendix A Personal Reflection ........................................................................ 45 Appendix B Ethical Issues Addressed .............................................................. 48 B. i: Ethical issues ....................................................................................... 48 Appendix C Record of External Materials Used ................................................ 49 Appendix D Full Rules of the Game ................................................................... 50 D. i: Components.......................................................................................... 50 D. ii: The Tiles .............................................................................................. 50 D. iii: Setup ................................................................................................... 50 D. iv: Playing the game................................................................................. 50 D. v: Winning the Game ............................................................................... 51 D. vi: FAQ..................................................................................................... 51 - vii Appendix E Project Code and Other Materials of Interest ............................... 52 E. i: coords.py .............................................................................................. 52 E. ii: vision.py ............................................................................................... 52 E. iii: Demonstration Video ........................................................................... 62 E. iv: GitHub Repository ............................................................................... 62 -1- Chapter 1 Introduction In this section of the report, I give the reader an overview of the project and the planning that went behind it. 1.1 Aims The aim of this project is to design and implement a software system for a Baxter robot to allow a human to play a simple board game with it. A secondary aim of this project is for me to learn more about the field of robotics and extend my knowledge of practical artificial intelligence, more specifically the application of game logic. 1.2 Objectives The main objectives for this project are as follows: I am to design or modify a game so that it is suitable for play with a Baxter robot. I am to implement a computer vision system to allow the Baxter robot to play with no prior knowledge of the game state or human input. I am to implement an artificial intelligence (AI) system to allow the robot to play competently. I am to implement it in such a way as to allow full game play with a human player. 1.3 Minimum Requirements The minimum requirements for this project are as follows: I am to design or modify a game so that it is suitable for play with a Baxter robot. I am to implement a system allowing the robot to play the game by itself. 1.4 The Problem In order for Lucas to play the game, there are several tasks it needs to perform in order to complete a game turn. Tile Manipulation: Lucas needs to be able to accurately pick tiles up and place them within a grid space -2 Tile identification: Lucas needs to be able to identify which type of tile it has just picked up. Board state identification: Lucas needs to know what tiles have been placed in each grid space, and what the implications of that would be on its potential moves. Game logic: Lucas needs to be able to make a decision as to how to place the tile based on the position of its piece and the possible paths on the tile. Turns and player interaction: Lucas needs to be able to indicate when its turn is over and be able to realise when its turn has come around again. Detect game end: Lucas needs to be able to detect when the game is over, either through all other players ejecting themselves from the board, or from Lucas sending his own path off of the board. In essence, this can be configured as a looping program with the game end being the break condition. The structure of a turn loop is as follows: 1 Wait to receive turn start signal o 2 Identify board state o 3 The tile is placed in a specific place on the table for Lucas to identify it. Identify the tile o 5 If the game has ended during another player’s turn, celebrate victory Draw a tile o 4 This is triggered when Lucas detects a new game piece in the draw space. Lucas needs to know which of the two types of tile it is. Decide how to place the tile o With the “straight ahead” tiles there is only one option, but the diagonal tiles can be placed in two different ways, going left or going right. 6 Move the tile to the appropriate grid square 7 Rotate the tile if necessary o As the diagonal tile has two possible outcomes (left or right), the tile may need to be rotated to direct the path in the correct direction. -3- 8 Place the tile 9 Check for game loss (sometimes a loss is unavoidable) o If Lucas has lost the game, be gracious in defeat 10 Send turn end signal 11 The loop begins anew. 1.5 Methodology Throughout this project I intended to use primarily a heavy-planning, heavy-testing agile modular approach to time management, with each section of the overall software solution being designed, developed and tested sequentially. As time passed and the full solution came together it became clear to me that some parts could not be tested without others. Pictured below is the initial schedule for the project, in the form of a Gantt Chart (Fig. 1.1). The general method I used for this project is to reduce each part of each task to its simplest form to produce a clear, well defined and functional program with minimal margin for error in execution. 1.6 Project Management As with any project, there are several things which must be taken into consideration, specifically when working with one particular robot. The most important risk to consider is the possibility of the robot breaking down and being non-functional for the delivery of the project. In this case the contingency plan would be to have the project running on simulation software rather than on the actual robot. Fortunately RethinkRobotics provides a Baxter package for Gazebo1, multi-robot simulation software which allows simulation of interaction with 3D environments. In the worst-case scenario (Lucas falls down some stairs or explodes or is otherwise incapacitated for the project delivery), this is the course of action to take. Other risks to consider would be risks to people near the robot, as the robot’s arms do not stop when they come into contact with something or someone. To minimise personal risk, no one should be within arm’s reach of the robot when it’s in operation, and the emergency stop 1 http://sdk.rethinkrobotics.com/wiki/Baxter_Simulator More information on the Gazebo simulator for use with Baxter robots. -4- button should always be to hand, although the inbuilt safety measures in the Baxter Robot should prevent anything grievous from happening. Just in case, however, the emergency shutdown button is always close at hand during operation. Due to Lucas’ tendency to attempt to move its arms through routes which include going through other game pieces or the table, the code for moving the arms is not as elegant as it could be, as this is compensated for by taking the arm more precisely through several small movements in each step as opposed to one large sweeping movement, which holds the potential for sowing chaos and destruction across the board to a minimum. -5- Figure 1.1: Gantt chart showing the initial planned schedule for the project -6- Chapter 2 Background And Literature Review In this section of the report, I provide some background knowledge on the aspects of computer science tackled in this project. 2.1 Game AI 2.1.1 Games From a Computing Perspective Russell and Norvig, in their Artificial Intelligence: A Modern Approach define games as: “competitive environments, in which the agents’ goals are in conflict, giving rise to adversarial search problems” (Russel, 2010) From a purely AI point of view, this makes perfect sense, and covers most of the points which define the kind of games that researchers are generally interested in programming robots to play, namely chess, backgammon and the game in this project. 2.1.2 Adversarial Searches and Pruning Adversarial searches function best in two playing, zero-sum games of perfect information such as chess, where everything is on the board and all of your opponents possible moves after your next move can be determined in advance. This can lead to some rather enormous search trees, especially in situations looking for optimal play. The term “Adversarial” comes from the fact that these are most often applied to competitive games, usually where a robot is attempting to best a human player, and the search algorithm will analyse possible moves from both players. This is epitomised by the “Minimax” algorithm which reads through possibilities for both players for several turns in advance, so called because it calculates the “max” or most optimal move for the agent, leading to the “min” or least optimal move for the opponent. As some games are so complex in nature that searching each individual possibility would take so long as to be impractical (chess, for example, is usually estimated to have around 35100 nodes in a maximal search tree) (Russel, 2010), and a lot of these moves can, at an early stage, be deemed to be purely detrimental to the state of play (such as a move which sends the player off of the board). These branches can be removed before too much time is used to search the rest of the tree, which can have a considerable impact as decision trees in game solving algorithms expand exponentially with each future turn explored. There are several factors which can influence the effectiveness of these search algorithms. They are most effective in situations where the agent has omniscience of the environment, in games such as chess or checkers where the entire board and all of the pieces which will ever -7- take place in the game are visible immediately. In the game presented in this project, however, future prediction through a minimax algorithm would be redundant, due to the random nature of the tile dealt, and the fact that there are a maximum of two decisions once a tile has been drawn. 2.1.2 Machine Learning Machine learning describes an aspect of computer science where a system learns or is taught how to evaluate data according to certain weighted parameters. For example, a system for filtering out news stories relevant to a particular subject could be taught by being fed a test dataset with the articles with the specific keywords to look for already marked out. From this, the machine learning algorithm can understand that the marked articles all contain something in common which is what the system is to look out for. For a game implementation, this would take the form of sets of moves and which moves lead to a more successful result. This type of machine learning is known as supervised learning. Supervised learning often uses Okham’s2 Razor (or lex parsimoniae) for deriving the shared attributes from a set of training data. Okham’s Razor states that when presented with several hypotheses that obtain the same result, the correct course of action would be to go with the simplest hypothesis which is the consistent with the data. (Russel, 2010) The other main method of machine learning implementation is reinforcement learning, where the agent performs the task multiple times and analyses the data to see how it can improve after each iteration. For example, in a machine learning algorithm for the game in this project, the set of moves performed by both players would be stored and the game would receive a value depending on how well the agent performed. Higher values would indicate a better performance, and the dataset could be analysed to see which moves in which board spaces lead to better results. Both methods of machine learning have their advantages and disadvantages, and often, especially in the field of games, semi-supervised learning is encountered, where a small training data set is provided, and then many un-labelled sets of data (such as playing a game repeatedly) are sorted. 2 Also seen spelled as “Occam’s” -8- 2.2 Machine Vision Machine vision describes the field of computer science used to allow automatic image analysis and inspection. This has many different applications including robot guidance and medical imaging. Many machine vision systems are designed to interpret images in real-time and feed the results to another system to process, such as automatic license plate scanners and robotic vision systems. Many machine vision systems make use of such techniques as edge or contour detection in identifying objects. These techniques make use of image analysis techniques and analyse pixels in contrast with the surrounding pixels to determine where edges are formed in the image. Contour detection works in a similar way but instead of simply returning which pixels are “edge pixels”, returns the contours of objects within the image. (Hornberg, 2007) Another approach taken for identifying objects is mask or template matching. In a template matching approach, the program is given a template of an object which is then used to detect any examples of that object in an image. Advanced variations of template matching also include rotations and scaling of the template, to detect the desired object within an image even if it is at an odd angle or depicted at a different size than the template model. 2.3 Robotics 2.3.1 Robotic Behaviour Architectures All robotics system act in accordance to a certain “paradigm”. The most traditional robotics paradigm is known as the sense-plan-act paradigm, which was used in some of the first autonomous robots, namely Shakey, a robot which explored and mapped an environment. (Nilsson, 1984) Sense-Plan-Act is structured in just the way that it sounds, a paradigm composed of three parts: 1: Sense – The agent examines the environment. 2: Plan – The agent takes this information and makes an informed decision on how to proceed. 3: Act – The agent undertakes the action decided through the “plan” step. -9- The most basic form of Sense-Plan-Act is the Hierarchical or Deliberative paradigm (depicted below in Fig 2.1), with a heavy focus on planning and gathering sensing data to update a global world model. Other models exist, which are primarily based on reactions to sensory triggers. Sense Plan Act Figure 2.1: Graphical representation of the Sense-Plan-Act Deliberative paradigm. Not long after the first practical implementations of Sense-Plan-Act, however, some researchers (namely Rodney Brooks, MIT), argued that sense-plan-act was “detrimental to the construction of real working robots” (Arkin, 1998) and advocated instead the use of a layered “subsumption” architecture based instead on a purely reactive behaviour-based model of operation. Although Arkin’s points on the subsumption architecture were, and remain, valid, in my opinion the architecture of the paradigm should be tailored to the intended purpose of the system. As such, due to the nature of the task in this project, a turn-based game, I think that the most suited paradigm for this is a traditional Deliberative Sense-Plan-Act paradigm, as at the start of a game turn each player gathers all of the necessary information, plans what to do with this information and then enacts the turn actions they choose to do. It seems to be the intuitive, logical fit, as opposed to convoluting the process with a reaction-based behaviour model.3 2.3.2 Inverse Kinematics The process involved in inverse kinematics usually begins with a kinematic analysis of the robots mechanical system, that is, an analysis of the lengths and ranges of the links between 3 There are a plethora of other robotic behaviour paradigms out there, but the most common comparison that seems to be drawn in academic literature is between these two, with Sense-Plan-Act being the one that I have deemed most relevant to this project. - 10 - each joint in each limb. Kinematics deals with how to move the limbs in a certain range and finding the end result, usually in terms of where the gripping mechanism at the end of the limb will end up. Inverse kinematics is the application of the reverse. Given a point in the environment, either in world coordinates or joint angles, inverse kinematic formulas are used in order to coordinate the limb to move from its current position to the given world position. There are several problems which often arise in inverse kinematics, the first and foremost being that there are multiple solutions to achieve the desired result, and the system has to be able to choose one. The criteria which is usually used in this decision is the solution that presents the least amount of movement while still achieving the desired final location. This ambiguity in the actual route the limb will take can lead to some unwanted effects, such as the limb taking an unexpected route through some solid objects in the environment, as the route which seems the closest to the robot may not be the route which would seem to be the closest to a human, as the robot will choose the closest solution in “joint-space” rather than our perspective of three-dimensional space (Craig, 2005). - 11 - Chapter 3 Materials In this chapter I go into detail about the design process for the game and the decisions made to make it suitable for Lucas and the project, as well as the decisions on how to interpret the game logic and which development tools to use in the project. 3.1 The Robot The robot used for this project is “Lucas”, the University Of Leeds School Of Computing’s Baxter robot, manufactured by Rethink Robotics. Baxter robots are composed of a torso containing their central processing system, a head with a screen for a face and two very mobile arms, each with 7 degrees of freedom with a camera and parallel grippers on the ends. Figure 3.1: Lucas in the laboratory. Notable features of the Baxter robots over other industrial robots include the manually controllable arms, which allows programs based around fixed positions (such as this project) to be implemented more easily, as well as rigorous safety protocols hard-coded into its operation, which prevent the arms from going through each other or surrounding objects.4 4 More information on RethinkRobotics’ http://www.rethinkrobotics.com/baxter/ Baxter robots can be found here: - 12 Baxter robots are produced primarily for use in manufacturing as an alternative to outsourcing labour, more specifically for monotonous tasks such as line loading and packaging. Notable features of the Baxter robots over other industrial robots include the manually controllable arms, as well as the rigorous safety protocols hard-coded into its operation. They have also found a place in many universities’ robotics laboratories due to the ease with which they can be programmed and the low risk to personal safety in working with one in comparison to other industrial robots. 3.2 The Game The game I designed for this project is heavily based on Tsuro: The Game of the Path5. Tsuro is played with a stack of tiles, a board, and one playing piece per player. The tiles each have 4 “paths” on them connecting 8 points. Gameplay starts by each player in turn placing their first tile along the edge of a board and choosing which path to take. In subsequent turns, the players continue placing tiles in such a way as to form a continuous path that their piece takes, attempting to ensure that their piece stays on the board and does not collide with other players’ pieces (as this links up their paths, both of which start on a board edge, thus resulting in both of their paths leading off of the board). A revised version of Tsuro, Tsuro Of The Seas, was released in 2012, however the additional mechanics added in simply add another element of chance into the game and as such have not been considered for this particular project. These added rules also detract from the simplicity and elegance which makes Tsuro so appealing, and Tsuro of the Seas is generally held in lower regard than the original. Figure 3.2: A two-player game of Tsuro in progress. 5 More information on Tsuro can https://boardgamegeek.com/boardgame/16992/tsuro be found here: - 13 - 3.2.1 Adapting the game to work with Lucas The game designed for this project is similar to – and inspired by - Tsuro, but simplified, with only 2 paths and 4 points per tile, thus meaning there are only two possible types of tile, as shown below in Fig 3.3: Figure 3.3: Schematic for the two tile designs used in the revised game, with connections to opposite sides on the left, and adjacent on the right. The pieces are custom made from wood, to appropriate dimensions for Lucas to be able to pick them up and manipulate them as needed, with electrical tape used for the colouring, due to its low reflectivity and to get consistent shades of black and white. The board is a 5 by 4 grid, a compromise between maintaining some complexity to the game, and the limited reach of Lucas’ arms and the space needed for the larger tiles, with some leeway included to accommodate for inaccuracy in placement, as well as to allow the arms of the parallel grippers to move between tiles. 3.2.2 Strategies There are two basic strategies in Tsuro, both of which can be developed upon, but the basic premises of both of which can be applied to the game in this project as well. The first of these is sticking to the edge (Fig 1.4): simply attempting to run around the edge of the board is a good way to keep space from tiles other players have placed, in an effort to “buy time” for yourself in order to survive longer in the game. However, with a small slip up or bad luck of the draw it is very easy to end up with no choice but to send yourself off of the edge. - 14 - Figure 3.4: Possible progression of play for a player attempting to stick to the edge of the board. The second strategy, the one which Lucas is programmed to follow in this project, is to stick as close to the centre as possible. There is a “golden zone” (as seen in Fig. 2.5) in the middle of the board where the player is at least 1 space away from the edge, and as such it becomes much harder to throw oneself off the board with a single bad play. This strategy essentially uses the outer regions of the board as “insurance”, and revolves around making sure that the player stays as far from the edges as possible. However this is risky as the squares in the centre of the grid are usually in high demand for these reasons. Figure 3.5: Showing the "Golden Zone" or "Safe Zone" in the centre of the board. Lucas will attempt to occupy these spaces for as much of the game as possible. - 15 - 3.3 Original Concept I had originally planned to program Lucas to play the full game of Tsuro, using pairings of coloured squares to define potential paths. However, after discussing this with my supervisors it became obvious that this was far too ambitious to achieve in the timescale for the project, due to the enormously increased level of complexity and challenge represented in almost every part of the project with the inclusion of 4 colour coded paths per tile. Figure 3.6: Early prototype game pieces for the original concept, comprised of Tsuro game pieces with path ends marked with coloured tape. 3.4 Dealing With Limitations Because the game had to be playable by a Baxter robot, there were several factors I had to take into consideration. The main change I made was reducing the number of potential paths on the block to 2, down from 4. This meant that, not only did I have to make custom pieces for the game, I had to redesign the game around this. 3.4.1 Vision Every tile that goes on the board needs to be scanned by Lucas, so that it can make a fully informed decision, as a human player would. As such the game pieces had to be made in such a way that they could be easily and, most importantly, accurately identified by a computer vision system. In order to make the blocks as easily identifiable as possible, the background of the blocks is black, with the paths shown in white, so that a computer vision system could reduce them to a binary image (where each pixel is black or white, no grey pixels) very easily with minimal - 16 - error. All three (from Lucas’ perspective) types of block are shown below in Fig. 3.76. Once reduced to a binary image, it would be much easier to determine the pattern on the block. Another factor affecting the vision element of the project is the size of the blocks. If the blocks were too small, it would be harder to obtain a clear image of the block face. Figure 3.7: The physical blocks used in the game. Block types (from left to right) 1, 2 and 3 shown here. The board is a black sheet of plywood, marked with an even 4x5 grid of squares, as shown below in Fig. 3.8 (although the board seems blotchy, the lighter areas are still dark enough to be classed as black in the binary image). The grid is marked in pencil, and as such is imperceptible to Lucas (especially after applying smoothing and filtering small areas from the image), but still easily visible to the human player. The board is black so as to make the identification of the patterns on the block as easy as possible, and to make all of the vision tasks easier and more accurate by providing a solid black background. 6 I will refer to the different block types by these designations throughout the course of the report, whereas for the pieces, I will use the words block, piece and tile interchangeably. - 17 - Figure 3.8: The game board, with the spaces marked and the grid labelled in pencil for the convenience of the human player. 3.4.2 Mechanical Baxter robots, while precise by modern standards of robotics, still have difficulty picking up objects thinner than a few centimetres with the electric parallel grippers. Each set of grippers also has a maximum and minimum width, which means the width of the game pieces has to be somewhere between these two values for the set of grippers that is to be used to manipulate them. Baxter robots can also tell when the grippers sense resistance, but may not stop immediately, or apply a certain amount of force before stopping. In order to avoid damage to the components through play, I had to make the blocks out of rigid material. Because of the slight potential for error in the arms placing the blocks and the fact that the grippers will have to fit between the blocks, the grid spaces need to be made to be larger than the blocks to accommodate this. As a result of these considerations, I made the game pieces as square blocks of wood, roughly 63mm square and 45mm deep. The size of the square face allows a large enough facing for the vision system to identify the block accurately, while also placing the block in the width range for the set of grippers used for manipulating the blocks to be able to go around the blocks and also grip them firmly. The blocks are covered in black electrical tape, with white electrical tape used to define the paths on each block. I chose electrical tape over painting the blocks as it gives a more even and defined coating of black and white than painting the blocks, and covers any errant splinters which may be apparent on the blocks. To allow the grippers to move smoothly between blocks on the grid, each of the grid squares is 80mm square. - 18 - 3.5 Consequences of Compromise on the Game With these compromises and the revised blocks in the game, this obviously has a significant impact on the rules and play of the game. The depth of the blocks mean that, rather than being picked from a stack (which would be almost a metre tall at the start of the game, far too cumbersome for Lucas to work around), the human player (or an unbiased 3rd party, to minimise the potential for foul play) has to deal blocks to Lucas on each of its turns. In simplifying the game, I had to tread a fine line between making it simple enough, while at the same time being careful not to remove the “game” from the exercise, simply placing blocks on a grid at random is merely an activity, there is no problem to solve. To maintain the game aspect, which is fundamental to this project, it is important that the final played game still fits within the “human” definition of a game, as well as the computational definition provided in 2.1.1. What exactly separates a game from an activity can seem to be obscure, so I have taken this definition to apply to the minimum standard of a game: “A game is a problem-solving activity, approached with a playful attitude” (Schell, 2015) While it may be hard for a robot to assume a “playful attitude”, I would assume the responsibility here would lie with the human player, although I intend to make Lucas act a bit more playfully than its usual solemn demeanour. 3.6 Breaking Down the Game Logic As detailed in chapter 1.4 The Problem, the game turn has a solid structure, which both human and robot players go through in typical play. The first stage of each turn is to identify the board state and recognise the move your opponent has made. After that, the player draws their own tile, which then has to be identified, before looking for the most optimal placement. Full rules for play can be found in Appendix C. A major feature in the game is that if the piece lands in such a way that the player’s path joins up with a tile already on the board, the path continues along it. This also applies if the opponent has placed their piece in the space your path advances to. As detailed in chapter 3.4, Lucas will be using a strategy which focuses on staying in the middle of the board. Lucas will also start the game by placing a tile on the edge of the board. The player’s path starts from the edge of the board and follows from there. There are several factors that have to be taken into account to understand the situation and make an informed decision: Current position: The board space the tile will be placed in. Current direction: The direction from which the space is being entered - 19 Tile type: This, in combination with the current position and direction gives the result of the move, or the next position and next direction, if you will. The next position and direction: This is where you end up after taking your turn. It may very well be off the board. If this is occupied, then this is not the final position and direction, and this needs to be followed up to find where the player is at the end of their turn. The basic turn structure in terms of thought processes and actions is displayed below in Figure 3.9 below. Figure 3.9: The structure of a turn in terms of thought processes and actions. Each of these points in the turn process can be further expanded upon, but to avoid repeating myself I’ll go into further detail in chapter 4 when I explain more about how the final program handles these actions. There are only 2 parts of this structure where any game logic actually needs to be applied, and they are somewhat intertwined. The decision on how to place a piece would, ultimately, depend on where the path leads to, as the decision would be based on whether one leads you off of the board, or into the central six spaces (according to the strategy which is used by Lucas in this project). In essence, these are the two factors that Lucas takes into account in its strategy when playing the game. Lucas will not take into account how its moves affect my future actions, or indeed how it will affect its actions in future turns. Although being taken off the board and being taken into the centre are mutually exclusive, the option to remain on the board at all costs is still treated with the highest priority. We can then assume that the decision trees would follow the logic depicted in Figure 3.10. - 20 - As per Figure 3.10, there are only two real outcomes from the turn after the piece is placed. Either the game is lost and the path leads off of the board or the move is good and the next space needs to be figured out. Figure 3.10: Flowchart depicting the logic involved in deciding how to place a tile and whether it results in game loss or finding the space for the next turn. - 21 - In order to ensure I had dissected the game logic thoroughly, I played around 50 games against humans and myself in order to make sure I had a firm grasp on all of the logic involved in decision making.7 3.7 Decisions on Development Tools I chose Python over C++ to be the language of choice for this project due to my familiarity with it (though I am quite experienced with C++ also), and because of the wide array of useful libraries and resources available for Python when working with C++. I also prefer Python personally as it is not as pernickety when it comes to variable types and passing variables. I also had several options to choose from when deciding how to handle the computer vision aspect of the implementation. All of the computer vision programming I had done before was in MATLAB, though I chose OpenCV because of the Python libraries available to interface between ROS and OpenCV (namely cv_bridge , which is invaluable in converting image types from the raw ROS source to a format which OpenCV can recognise). For version control, I chose to use a GitHub git repository as, when faced with the options available, I found that GitHub had all of the features I needed, as well as being easier to set up, commit to, and retrieve files from remotely than the alternatives. I deemed version control (as well as a repository) to be a necessity for this project, as in order to write this report I would need access to the code wherever I am (which GitHub allows through a browser interface for extra convenience), and keeping track of features over various incarnations of the project is useful in case a particular “fix” or “improvement” update turns out to be quite detrimental and needs to be undone. 7 I have also played several dozen games of Tsuro in the past. - 22 - Chapter 4 Methods In this chapter of the report, I go into detail about the design, development and operation of the Software project. 4.1 The Vision System There are two functions in the program which form the two distinct parts to the vision element of this project: IDBlock(img): This function takes an image from the left-hand camera of the space underneath the arm at that point, and identifies which type of game piece is in that space. ScanBoard(img): This function takes an image from the right-hand camera of an orthogonal view of the board, and scans each grid space for changes. Once the space in which a tile has been placed is identified, it calls for the right arm to move to that space and calls IDBlock(img) Both of these functions start by using thresholding to convert each pixel to be either black or white (a “binary image”), depending on which it is closest to, with the components all being black or white, this reduces noise and essentially makes sure that Lucas is seeing the environment as clearly defined pieces. 4.1.1 Block Identification After converting the image to binary, to reduce the capacity for error the image is cropped to the area containing the desired block that needs identification. This stage of the process is vital as the full original image contains segments of up to 6 board spaces. The cropped image is then searched for shapes using OpenCV’s findContours function, which searches for continuous areas of white and stores outlines of them. In case any small contours have been drawn around noise anomalies, only contours with an area greater than a certain thresholding value are kept and used. Using an approach based on taking only the essential data required to successfully and accurately identify a block, the most basic set of assumptions we can make to differentiate between the different types of blocks, while still having complete accuracy is as follows: The type 1 block has one contour (the cross), and the type 2 and 3 blocks have 2 contours (the two diagonal paths). To differentiate between type 2 and 3, the simplest way to tell which is which is to locate the centre points of each of the diagonals, and compare them to each other. - 23 If one of the points has both a greater x and y value than the other, then the block is type 2, otherwise it is type 3. The entire process can be seen below over the course of figures 4.1.1 to 4.1.4: Step 1: Lucas captures an image of the area the arm is currently above. Figure 4.1.1: The image captured by Lucas when positioned over the desired block. Step 2: Lucas crops the image to the area containing the desired game piece and converts the image to binary format. Figure 4.1.2: The image has been cropped to the right area and converted to a binary image. - 24 Step 3: Lucas, using the findContours function, identifies all of the areas of continuous white in the image (they have been coloured in green here). Figure 4.1.3: The contours of the areas of white have been located. Step 4: Lucas filters the contours to only include those big enough to be relevant, identifies that there are two of them and draws a line between them. The two points of the line are then compared to see if it is a type 2 or type 3 piece. Figure 4.1.4: The relevant contours have been found and the central points are compared. In this case, the function would return that this piece is a "type 2" piece. - 25 - Another approach that I considered and explored was to find the contour of the entire block face. Once this was identified, I could attempt to deduce the type of block by comparing the amount of black with the amount of white in the central third of the block. If the area was mostly white, it was a type 1 block, as the two paths cross in the centre. If the area was mostly black, it was a type 2 or 3 block, as both paths on those blocks avoid the centre. I would then have been able to find the specific type of “diagonal” block by comparing the corners of this central area to each other. I abandoned this method while designing the vision system as, due to inconsistencies in the blocks and the difficulty involved in finding the contour of a two-colour object, it was too unreliable and added unnecessary complications, especially when contrasted with the final method used, detailed above. IDBlock() also functions as the trigger to start a new turn. At the end of the previous turn, Lucas begins checking the Pickup spot for a new piece at short intervals. When this becomes non-zero as Lucas is dealt a new piece, the turn begins and Lucas then scans the board. 4.1.2 Scanning the Board State The method I devised for scanning the board and identifying the current board state at the start of each turn works as follows: At the start of each turn, move the right arm camera to provide an orthogonal view of the board. Capture an image of the entire board area. The right arm can then be tucked away so as to not obstruct the left arm. For each space that is still marked as empty within the game logic system, scan the corresponding area of the image. If there is a change, move the right arm to that space and run IDBlock(img) on that area. As only one space per turn would change (Lucas marks the tiles it places down as they are placed), Lucas can then move onto the next part of the turn. This method, while slow, is accurate and very reliable. The image is captured, converted to binary using thresholding and stored. For each empty board space remaining, a version of the entire image of the board is cropped to fit that space. If the average pixel intensity, found using numpy.mean on the cropped image (which takes advantage of the fact that OpenCV treats images as arrays), in the space is greater than a certain value (Around 0.4, with 0 being completely black and 1 being completely white), the value for the space and the space designation are stored. The space in this list with the highest average pixel intensity is then chosen and the left arm is sent to that space and IDBlock(img) is run to detect which kind of block has been placed there. - 26 - Figure 4.2: The Robot marks the space which is marked as empty that it has perceived to have a piece in from the pixel intensity of the space. 4.2 The Mechanical System There are three main methods to control the arms of a Baxter robot: World coordinates. Joint angles. A vision-based system. In a world coordinates-based system, the robot navigates the environment by moving the arms to certain positions defined as (x, y, z) coordinates in the area around it. These can be calculated given the origin and relative positions of objects and points in the environment. In a joint-angles based system, the robot moves an arm between positions by moving each of the motors in the arm to a specific pre-set position. These can be gotten from Lucas by placing the arm in each required position and requesting them through a Python shell. In essence, these two methods do not differ very much, as they both involve moving the arm to a predefined destination point without having a clear set path. This means that occasionally the arm may take an unexpected path which could potentially try to lead the arm through something solid. This was encountered in development of an early demo version of the project - 27 - where Lucas was attempting to take a piece from the top of a stack but the path it attempted to take to that point in the environment was taking the arm through the stack of tiles. The rather inelegant solution to this problem is to set more positions at small intervals between the points, forcing the arm to travel in small, easily predictable steps, rather than sweeping gestures. Each of the two systems detailed above has its advantages and disadvantages, and I took these into consideration when choosing which to use in the project. In the end I decided to use joint angles to determine the positions, as it is much easier to gather the positions from the hardware in this way than to figure out the 45 world coordinates I needed. Due to the difficulties mentioned in 4.1.2 with getting contours for entire blocks, I did not attempt to investigate a vision-based system where the robot identifies objects and moves the arm autonomously to pick it up, as it would have had considerable difficulty in identifying where the block is exactly, as well as which block to pick up. Vision-based systems work best when Lucas is needed to pick up distinctly coloured objects from unpredictable positions, whereas many of the game pieces are identical and will always need to be picked up from or placed in exact, pre-decided positions. It is also possible to program a fully pre-set path through a toolkit provided by RethinkRobotics, but I didn’t investigate this course of action very far as it seemed to be unintuitive and difficult to integrate with the rest of the project. Each movement in the game turn has a specific structure to minimise the opportunity for disruption caused by unpredictable movement paths. There is a “Reset” point far above the board, and each space on the board has two sets of joint angles defined in the program to determine the path for the arm to take to the point, designated as (space)U and (space)D, e.g. A1U and A1D. As an example, the route the arm would take to pick up a new block and place it in space C3 would be as shown below in Fig. 4.3: Figure 4.3: Flowchart depicting the orders that the left arm goes through in picking up a block and taking it to a certain board space. - 28 - 4.3 The Game Logic System To program the game logic into the implementation of this project, for the decision making part, I took a simple approach of a decision tree comprised of nested IF statements describing the decision logic depicted in the flowchart in Fig 3.8, resulting in the next space (defined by the global variable next_space) being assigned to the space which either: For a type 1 game piece, was the next block in that direction For a type 2 or 3 piece, was the choice which was: o If possible, in the centre 6 spaces. o If not, then, if possible, was not off of the board. If the space was off of the board, it was defined as space 668 The main decision tree is contained in the turn loop itself, as the positioning of it within the loop is of absolute importance. One important factor I will talk about in more detail in section 4.4 is that each board space is represented in the code as an object (an instance of the BoardSpace class defined in the code), and stored in a Python list (board_spaces ). The board spaces are referred to throughout the code almost exclusively as the integer that defines them as a member of this list, so the global and local constants that refer to board spaces are usually integers, and the functions usually operate on these integers which are later used to retrieve more information about the particular board space(s) (though there are exceptions to this). There are two main functions which comprise the majority of the rest of the game logic: nextSpace() which, given the current direction, assigns the next space as an arithmetic operator on the current space, and checks to see if this results in the path leading off of the board. If the path leads off the board it re-assigns it to a value of 66 changeDirection() which takes the direction of travel at the start of the turn, and figures out the new direction based on the piece placed in the turn. As an example, a direction of “up” and current_piece = 2 would result in the direction being changed to “right” While programming the logic I had to take certain factors into account. In an earlier version of the program, changeDirection() was integrated into nextSpace(), which, when the next space was calculated for both options when placing a 2 or 3 type piece, calculated the change 8 I chose a value of 66 arbitrarily because it was well outside of the range of possibly assigned values for current_space . - 29 - in direction twice, leading to some very interesting and very illegal game moves on the part of Lucas. The nextSpace() function works around the premise of the board_space list and the current_space and next_space variables acting as references to its indices. The list stores BoardSpace objects referring to each space, starting with space A1 (the top left space) in list position 0, going across each of the rows of the board and finishing with space D5 in list position 20. Because of the structure of this list, moving around the board in terms of the list elements is incredibly easy. To move one space up on the board: o next_space = current_space – 5 o e.g. from B3 to A3 To move one space down on the board: o next_space = current_space + 5 o e.g. from B3 to C3 To move one square to the right: o next_space = current_space + 1 o e.g. from D2 to D3 To move one space to the left: o next_space = current_space – 1 o e.g. from B5 to B4 This function is also where the first check for game loss takes place, when the move is calculated and, if it takes the path off of the board, the current_space variable is assigned to 66. There are two different methods used to check depending on the current direction. For vertical travel, it simply checks if the current_space value leaves the list index range. o E.g. moving downwards from space D4 (list index 18) would result in a current_space value of 23, which would be reassigned to 66. For horizontal travel, it checks the direction of travel and the board space along the respective edges. o E.g. moving left from space C1 (list index 10) would result in a current_space of 9, however since the direction was “left” and the space was “10”, the move leads off of the board, which results in current_space being reassigned to 66. During the early stages of this project’s conception, I considered implementing a machine learning algorithm or some form of alpha-beta pruning (as described in Section 2.1.2). However, as the game was analysed and the features in it were scaled back it became - 30 - apparent that these were unnecessary complexities due to the random dealing nature of the game, and that the only real choice is which direction to go in when given a type 2 or 3 tile. If time allowed, I would have implemented the option to give the players a hand of 2 pieces to choose from, which may have warranted some advanced AI features such as these given the increase in game complexity.9 4.4 Putting It All Together I found the trickiest part of the implementation was what came after making all of the various pieces of the puzzle separately, the vision systems, the mechanical system and the game logic, and, essentially, shoving it all together and seeing if it works. In this part of the implementation I ended up changing a lot with how the global variables are handled by all of the different functions, as when I had originally written the functions, some of them took BoardSpace objects as variables when they only needed the current_space integer to perform. The BoardSpace class is a simple object class that stores a total of 7 variables about each board space: The “Up” position joint angles, for moving the arm to a point above that board space. The “Down” position joint angles, for moving the arm to the board space proper. The “piece” integer variable defining which piece is occupying that board space. Four integers corresponding to x and y coordinates describing the area occupied byu this board space in the image of the orthographic view of the board used by the scanBoard() function. When I was testing each stage of the combined implementation, at some points I also realised that I needed to put in more functions to manage the global variables to stop confusion (such as with the issue with the direction variable mentioned in 4.3) as well as refine the mechanical system (moving back from identifying a tile would take a direct path to the reset point, instead of traveling through the interval point above the board space, which occasionally disrupted other board pieces). The final implementation uses a total of 14 functions, which often refer to each other and are all utilised in the course of a game. 9 Unfortunately I did not get around to doing this, though I will discuss this further in chapter 6.2. - 31 - The vision-based functions are: get_right(msg) and get_left(msg) which take the images from the corresponding arm cameras and convert them into a format useable by OpenCV. IDBlock(img) which uses an image to decipher what type of game piece it is currently looking at. Explained in more detail in section 4.1.1. scanBoard(img) which scans the board for changes before identifying any spaces where a block has been placed and using IDBlock and moving the left arm to identify the block in that space. Explained in more detail in section 4.1.2. The mechanical functions are: place(num) which places a block in the board space at index num in board_spaces move_to(bs) and move_from(bs), which move the left arm to and from a certain board space respectively. Used in boardScan() to move to the block which needs to be identified with IDBlock(). pickUp() which picks up the new block from the draw space and identifies it with IDBlock(). rotate() which picks up and rotates the block on the draw space before calling pickUp() to pick it back up and re-identify it. And finally, the logical functions: startTurn() which instructs Lucas to wait for a piece to be placed in the draw space before its turn starts. This also acts as the trigger for detecting victory, if Lucas waits more than 30 seconds for a piece, it assumes it has won and the game is over. This is when scanBoard() is called. nextSpace(piece) given a piece type and the current space and direction, this function calculates the next space that this path would call it to. changeDirection() which applies the change in direction implied by the current piece. turnOne() which instructs Lucas to make a particular move in the first turn. This move is the safest move as it places a tile in the centre of one of the longer board edges, moving straight towards the central point of the board. boardStatePrint() is not a logical function, but prints a grid depicting the board state Lucas currently has stored. I put this in more for testing purposes than anything else. There is also the final game loop, which follows directly on from a call of the turnOne() function. This loop is a while loop which ends when the Boolean gameLoss stops being False, and is formed of three main parts. - 32 - The start-of-turn board state inspection and deduction is the first of these parts. In this stage, startTurn() is called, which waits for the new tile or declares victory, and triggers the boardScan() function to run. After this, if there is a piece in the space which Lucas is about to play into, it tracks this path to its end. This is done with the snippet of code shown below in Figure 4.4: Figure 4.4: The if and while statements used to track the current path through to the end and trigger gameLoss if the path leads off of the board. The main decision tree and its result from the second part of the turn. In this stage, Lucas uses the logic described in sections 3.6 and 4.3 to decide how to place a tile and see the resulting next_space where it will be placing a tile in its next turn. As described in section 4.3, this decision tree is a direct translation of the flowchart in figure 3.8. This includes rotating the piece if necessary and placing it on the appropriate space. Each turn ends with a short roundup step after the decision tree is executed. The direction is updated with changeDirection() and boardStatePrint() is called to print the current stored board state into the console. If any of the triggers that cause Lucas to register that it has lost the game go off, gameLost will return True, which will break the loop and cause the final part of the program to trigger. When this happens, Lucas “refuses to believe” that a human could have bested it and returns the following message: “+++Divide By Cucumber Error. Please Reinstall Universe And Reboot+++”10 10 This is a reference to HEX, a fictional, magical computer originally powered by ants navigating through glass tubes before evolving into something much more magnificent (because, well, magic), from the Discworld of the late, great Sir Terry Pratchett, from whose work I have gained an appreciation for all things exciting and out of the ordinary, and a penchant for going off on a tangent in footnotes. - 33 - Chapter 5 Analysis and Evaluation of the System In this chapter I critically analyse the reliability and performance of the system as a whole, and address any issues which I observed. I also provide, analyse and discuss the results from game data gathered and evaluate the decisions I made at the start, and throughout, the project. 5.1 System Reliability The system in its final incarnation proved to be very reliable, with only one game of the 20 having to be exited early due to a misinterpretation of a tile leading Lucas to make an illegal move (although not from its perspective). The grippers never dropped a tile in mid-air and the hardware performed consistently. The game components served their purpose absolutely and without error or causing or withstanding any damage to themselves, the robot or the environment throughout the course of the testing and development. There was only one major bug which manifested repeatedly in testing, and this is due to the fact that I misallocated the value of “B1U” in “coords.py” as the same value as “A5U”.11 This leads to Lucas moving its arm (and any block it happens to be holding) rapidly across the board at board level, scattering any blocks in its path. An attentive human player can catch this as it happens and minimise the damage caused by replacing the scattered blocks (if any). It is interesting to note, however, that this has absolutely no effect on Lucas’ play, as it has already allocated values for those board spaces of the correct tile, and will not scan them in the scanBoard() function, and will make decisions based on the tiles that were placed there. Replacing the tiles instead of removing them from the board then is only in the interest of the human player and the program is quite indifferent, because, as far as it is concerned, those board spaces have game pieces in them. One other minor unintended negative effect is that, if Lucas is attempting to identify a block in a space with other blocks placed horizontally adjacent, quite often the path of the grippers will be on path to collide with one or both of the blocks placed either side of the piece intended to be identified. Due to the fact that two solid objects cannot occupy the same point in space at the same time, the blocks either side must be moved slightly to allow the arm to descend into the required position for the IDBlock() function to be run properly. If the blocks are moved after the grippers have made contact, the joints in Lucas’ arms will apply excess pressure, and when the blocks are removed, will move too far down and may collide with the table, so 11 I have since fixed this by assigning a correct set of joint angles to coords.B1U. - 34 - vigilance on this matter is important. This could only be resolved by making a board with larger spaces, at which point the arm may not be able to reach all of the spaces on the board, or by playing with smaller pieces and installing smaller grippers on the arm, which would increase the precision required in the vision and mechanical systems. 5.2 Effectiveness The system proved relatively effective, with a win rate of 35%, a draw rate of 5% and an error prompted exit rate of 5%. The strategy used by Lucas was simple, but effective. The major downfall in its strategy is that it does not take into account its opponents future moves, or the moves it will have to make after the next round of play. I, as the human player, took full advantage of further forward planning and knowing how my moves would affect Lucas’ turn. This worked against the overall win rate of the system and gave me a clear advantage. However, the aim of this project was not to produce a system that wins every time (especially difficult if not impossible due to the randomised nature of the game), but to produce a system that interacts with a human through the medium of a game and plays sensible moves that stick to a specific strategy. Using this definition of effectiveness for the system, valuation is subjective along with the term “sensible”, however I would say that the robot clearly and consistently executed sensible game moves, never ejecting itself from the board if it had a choice, and consistently staying with the strategy detailed in sections 3.2.2 and 3.6 as much as the tiles dealt allowed it to, although several times Lucas boxed itself into a corner where the alternative choice would have been better (as shown in Figure 5.1 below), but this was not accounted for in the logic programmed in the decision tree. Figure 5.110: An example of an objectively illogical move which would be made by Lucas. By going to the right, Lucas only has one space free and will lose if dealt a type 1 tile in the next round. By going left instead, Lucas would have gained much more space and a better chance at survival - 35 - There is one exception to this, due to the lack of foresight in the space calculations in the decision tree. In hindsight it’s something which was easy to overlook, and relatively simple to implement, but the fact that Lucas placed a tile which joined both of our paths together, leading to a draw in test game 13 instead of the alternative option which would have kept its path on the board while sending mine off. The decision loop only takes into account the next space. It doesn’t investigate as to whether that space has a tile in it or where that path leads. That is not calculated in the game loop until the start of the next turn, by which point the human player has conceded defeat (as their path leads off of the board) and Lucas therefore assumes victory. Each game took on average just over a minute per game round, with an average game lasting between 6 and 10 minutes. I am quite pleased with the (relatively) swift execution of Lucas’ turns, considering how much arm movement is involved in each game turn, as well as the number of individual image processing functions called over the course of a game. 5.2.1 Effectiveness of the Vision Systems In the 20 games from which data was gathered, the vision systems were very effective, with only 8 misidentification errors over a total of 254 IDBlock() function calls and 1 error over 127 scanBoard() function calls. This gives the combined vision systems an overall error rate of only 2.36%. Misidentification errors were detected by comparing the final board state printed by Lucas with the actual configuration of tiles on the board at the end of the game. 5.2.2 Effectiveness of the Mechanical Systems With the exception of the aforementioned “B1U” bug (Section 5.1), which has since been fixed and had no real impact on gameplay from Lucas’ perspective, the mechanical systems performed as predicted. I am especially pleased with the effectiveness of the rotate() function, as some of the blocks are not perfectly square, but Lucas nevertheless picked them up neatly once rotated. Over the course of the test games, Lucas never dropped a game piece mid-air or performed any drastic moves which caused the board to shift on the table, both of which had occasionally been an issue during development. 5.3 Limitations Imposed By the System There are several limitations that this implementation imposes, namely the fact that a lot of the time the game ends due to the fact that the player had no choice about how to proceed, for example, getting dealt several type 1 pieces in a row, leading straight off the edge of the board. This is an issue more with the game than the software implementation though. - 36 - Lucas will only start in one of two spaces, and will only choose one if the other has already been taken up by the human player. In addition, Lucas will always start with a type 1 tile. This makes its opening moves fairly predictable. In order to ensure complete fairness as well (although I dealt tiles to both players randomly in testing), a third party is required to deal tiles to both players. Failing this, tiles need to be shuffled and placed in separate, randomly ordered stacks of 10, one stack for each player. Due to the fixed nature of the joint angles in terms of manoeuvring around the environment, the board must be placed in exactly the same place every time when playing with Lucas. In addition to this, due to the materials used for the components and the way the vision system operates, the vision systems only operate reliably in a dimly lit environment, with no direct lighting on the game board. 5.4 Gameplay Data All of the data gathered from the 20 games is shown in table 5-1, with a key to the Issues/Bugs column of data provided in table 5-1a. Table 5-1a: Key to Issues/Bugs reported in table 5-2 Ob Obstruction. Piece had to be moved to allow grippers to pass so that Lucas could ID a block placed on the board MI(x) MisIdentified. Lucas misidentified a block it picked up or looked at x times in the course of the game. The game was continued so long as this had no impact on Lucas’ play. DNR Did Not Recognise. Lucas did not recognise where a block had been placed and ran IDBlock on an empty space/no space at all IM Illegal Move. Other factors led to Lucas playing an illegal move. - 37 - Table 5-1: Results of the 20 games played against Lucas on the final version of the project Game Number Number of Lucas’ Result for Lucas Issues/Bugs Turns 1 5 Loss 2 7 Win 3 6 Loss 4 6 N/A 5 6 Loss 6 6 Win 7 8 Loss 8 6 Loss 9 5 Win 10 7 Loss 11 6 Win 12 7 Loss 13 7 Draw 14 6 Win MI(1) 15 5 Loss MI(2) 16 8 Win Ob 17 6 Loss 18 7 Loss Ob 19 6 Win Ob 20 7 Loss Ob Ob, MI(1) Ob Ob, MI(2), DNR -> IM Ob Ob, MI(1) Ob MI(1) Ob - 38 - 5.5 Discussion of Results 5.5.1 Data Gathering Methods Over the course of the testing stage, before any data was gathered, I decided that the only results that were of import to the overall conclusion were the number of turns taken by Lucas, the game result for Lucas and any issues or bugs that arose during the game. For the gathering of results, I was the human player and dealt tiles to both of us from a randomised face-down piles. In an ideal situation the human player would be someone who did not know the exact logic behind Lucas’ actions, and there would have been an impartial third party dealing tiles to each player. I attempted to make play decisions as if I were facing a human player, and blindly selected tiles at random from the stack. During the data gathering stage, the first turn was taken alternately by Lucas and the human player. 5.5.2 Data Analysis A first-glance analysis of the data gathered returns this information: The mean number of turns taken by Lucas in a game was 6.35. The modal number of turns taken by Lucas in a game was 6. Lucas was slightly more likely to win when going first, with 4 of its 7 victories occurring on even numbered games (when Lucas took the first turn). The mean number of turns in games where Lucas won was 6.29. Lucas only won 2 of the 8 games which lasted 7 turns or more. The most common issue that arose was obstruction, where one or more blocks had to be moved to allow the grippers to pass. The majority of the time, misidentification of a tile placed by the human player did not have any effect on Lucas’ play. 5.6 Evaluation of the Project 5.6.1 Project Schedule After the first few weeks of work on the project it became evident that the original schedule was not adhered to and it was disregarded from then on. At this point, time allocation became much more flexible, and I invested a lot of time into meticulous research on the programming methods and hardware I was going to be using in the project, as well as drawing up full pseudo-code prototypes for the implementations, none of which exactly resemble the final product. Once development actually began in the latter half of the project timeframe, the Gantt chart became a useful resource for tracking milestones and acting as a flowchart for devising which parts of the project should be implemented before others. - 39 - 5.6.2 Choice of Development Tools I feel that the choices I made in terms of development tools were optimal on every front. The lenience in passing and handling variables in Python was invaluable in a system where the main object class was comprised of two arrays where each of the entries was comprised of both a string and a float, and four integer values, and the main form of accessing and manipulating these objects was through a list. Attempting this in C++ would have been much harder and probably would have required a different and more convoluted approach. OpenCV, once I had some experience with the library and the unusual parameters involved in its function calls, was perfectly suited to the tasks I required it to do with each task I needed it for being solved in a simple function call or object reassignment. GitHub was the perfect platform for version control, allowing me to edit the code on the repository from home, through my web browser, whenever I thought of a small fix or feature to apply (or comment in in the case of larger edits which would have required testing) without having to install or download anything to my home computer. 5.6.3 Decisions in Game and Component Design The game design is solid, easy to understand and one of the things I am most proud of in this project. It is not too complex for this project nor is it too simple to meet the requirements. All of the pieces required for the game can be represented as chunky blocks that the robot can pick up and the designs on them are large enough to be easily recognised by the system. However, in hindsight, electrical tape was not the best choice for the blocks. This is due to the reflectivity of the black tape which, while not as reflective as other tapes or certain types of paint, will reflect what appears to a computer vision system as white light if under a direct light source. This restricts the game to being played under a suitable lighting condition. In retrospect, a printed or cut-out matte paper covering would have been the optimal material choice to represent the patterns on the game pieces. 5.6.4 Decisions in Software Design I am confident in the methods I used for the software design. They all follow the approach I set out with when starting this implementation which is to reduce each factor to its minimum parts to reduce the margin for error (for example, instead of using template matching to identify blocks, simply counting the number of white areas on the block). This has resulted in a set of complimentary vision systems with minimal error rates and a solid, logical decision tree. Each of the functions in the finished project serves a purpose, and no computation time is wasted on irrelevant or superfluous tasks. - 40 - Chapter 6 Conclusions In which I provide a closing statement on the final state of the project, conclude whether or not I consider it to be a success, and suggest ways in which it could be expanded given the time and resources. 6.1 Conclusions In the conclusion of this report, there are several questions which I have to ask myself about the project, in order to deem whether or not it was a success. “Was the project successful?” is the most obvious question, however there are several factors that have to be addressed and analysed before success of the project can be established. In order to deduce success, we must look at the aims and objectives that I set for this project to achieve. Firstly, the minimum requirements for the project were as follows: I am to design or modify a game so that it is suitable for play with a Baxter robot. I am to implement a system allowing the robot to play the game by itself. Both of these have been comfortably met. The data shows that the game is suitable to the task and, in addition to this, the game meets both definitions of what makes a game presented in this project. Lucas can play the game by itself if it is continually dealt tiles. It will simply attempt to stay on the board as long as possible with the tiles given. Next we need to look at the main objectives: I am to design or modify a game so that it is suitable for play with a Baxter robot. I am to implement a computer vision system to allow the Baxter robot to play with no prior knowledge of the game state or human input. I am to implement an artificial intelligence (AI) system to allow the robot to play competently. I am to implement it in such a way as to allow full game play with a human player. The game is fit for play with a Baxter robot, as is made evident by the fact that 20 full games were played. The game is a modified version of Tsuro by my own design, simplifying the game pieces and modifying the game rules to suit the new pieces. The final design was partially influenced by suggestions from my supervisors and from observations made during development. - 41 - The computer vision systems allow Lucas to play completely autonomously without any human input, with only a 2.36% rate of error over 381 separate vision function calls, which only once over the course of 20 games resulted in the game having to be abandoned. The AI system results in sensible play, although this is a subjective assessment. While playing it felt to me very much like a child playing chess with their parent. Lucas was definitely playing the game independently and making decisions based on the current board state but without considering the future implications of its play or how its play may affect the other player. Even with the only decision in the game being where to start and how to place tiles which are not of type 1, there are a few factors that are not considered in the AI. All things considered I think that the AI for play is competent, but has a lot of room for improvement. The final objective is that the full implementation allows play of a full game with a human player. Having played 20 games with Lucas in full with the test version of the program running, I can wholeheartedly conclude that play of a full game to completion is possible, and that this objective has been thoroughly met. In conclusion then: Was this project a success? I believe that, overall, taking all of the results and components into account, this project was a success, (especially since the bugs present during testing and in the demonstration video12 although there is a lot of room for expansion, especially given the subjective nature of a lot of the evaluative criteria (notably being “competent” at a game, as well as making “sensible” moves). The implementation is still far from perfect, some of the issues detailed in Chapter 5 still occur, especially the obstruction of the arm movement by other blocks on the board blocking the grippers, which is a fairly regular occurrence. The test version of the project derailed one of the 20 games wildly by making an illegal play after having misidentified a block. I theorise that most of the block misidentifications were due to the way the block was placed by the human player, as the image is cropped to a narrow window with little margin for error in IDBlock() and the functionality of IDBlock() can be compromised if the block is placed half an inch out from the ideal position. The system is inflexible and too precise for its own good, but despite its faults, the vast majority of the time works perfectly as intended. The final program is still dependent on the human player placing a block for the robot to draw each turn, and does not communicate with the player directly, however I think that the solutions to these lie outside of the basic scope of the project, although they definitely fit the theme of the project and would be worthwhile expansions. The secondary aim of this project was for me to gain a deeper understanding of the field of robotics and extend my knowledge of practical artificial intelligence, more specifically the 12 A link to which can be found in Appendix E - 42 - application of game logic. I can attest that throughout the course of this project, from my initial research into applied game logic systems through to the final stages of development and testing, I was constantly learning new applications and methods of implementation for the fundamental logic algorithms in game AI. I have also learnt a significant amount about practical robotics – a subject which I knew very little about at the start of this project – and applied it to achieve all of the goals and objectives set for this project. 6.2 Future Work There are a multitude of ways in which this project could be expanded, given more time and/or resources. Expansions that could be implemented without modifying the game include: A more advanced game AI system, allowing Lucas to predict the possible implications of its moves on the future moves of both it and its opponent. o A minimax algorithm could be implemented here to also encourage a more competitive style of play. o A machine learning algorithm could also be implemented, to allow Lucas to form its own strategy based on how placing tiles in various ways in certain spaces leads to a better or worse result. Implementing a separate array and a second global direction variable to keep track of its opponents moves and be able to detect when it has won through inspecting the board state, rather than the current method of assuming game loss when its piece is not renewed. Instead of detecting the start of the turn by using IDBlock() and waiting for a new tile to be dealt, detecting a change in board state (which occurs when the other player places a tile) by using boardScan() at short intervals to trigger the start of the turn. Lucas could be made to draw a new tile itself when it receives a start of turn trigger, from a stack of randomised game pieces, negating the need for an impartial dealer. Using an expanded vision system to detect where the board grid is, and moving to the various points defined according to this vision-based perspective rather than the - 43 - mechanical world perspective currently used. This also negates the need for the board to be placed in the exact same position for every game. Allowing play with multiple human players, and scanning each change in board state between turns as opposed to only one space. Game AI could also be expanded for this, taking more variables into consideration with the increased number of players. Using a randomised starting space and tile. At the moment, in the first turn, Lucas will only place in one of two spaces, and always uses a type 1 piece in its first turn. With improvements to the vision system (and a stable lighting environment), it would be possible to implement the full game of Tsuro, with 4 colour-coordinated paths to each tile. Each of the paths would be represented with a pair of matching coloured points (as seen in Fig 7.1). This would also add another layer of complexity in the choice, as each side of each tile has two paths linked, meaning there is a possible 7 exit paths. Game complexity could be further increased by also giving Lucas a choice of 3 tiles, bringing it up to a full implementation of the original Tsuro. With the game being made so much more complex, it would require more complex AI as well, rather than just a which would make use of complex decision trees and alpha-beta pruning. This could also be expanded upon with the application of a machine learning algorithm, gathering data based on which tiles leads to a greater chance of success, or which tile placements lead to the opponent careening off the board. Of course, it would also be possible to play different games using a modified version of this system. With a sufficiently developed vision system and a much more complicated artificial intelligence system, I believe it would even be possible to implement it so that Lucas could play chess against a human player using similar principles to form the base of the vision and mechanical systems – and assuming the pieces are blocks of the right size. To develop more on the human interaction side of this project, I would look at implementing a speech input system and text-to-speech system to allow certain phrases to be “spoken” between the player and the robot, mostly phrases such as “let’s play”, “your turn” or “good game, well played” which can obviously be interpreted as triggers for events in the course of the game, such as a turn ending or victory being declared by a player. - 44 - List of References Arkin, R. C., 1998. Behavior-based Robotics. Cambridge (US-MA): MIT Press. Craig, J. J., 2005. Introduction to Robotics - Mechanics and Control. 3rd ed. Upper Saddle River, NJ.: Pearson Education Inc.. Hornberg, A., 2007. Handbook of Machine Vision. 1st ed. Weinheim: John Wiley & Sons. Nilsson, N. J., 1984. Shakey The Robot, Technical Note 323. Menlo Park: SRI International. Russel, S. &. N. P., 2010. Artificial Intelligence: A Modern Approach. 3rd ed. Upper Saddle River: Pearson Education Inc.. Schell, J., 2015. The Art of Game Design. 2nd ed. Boca Raton, FL: CRC Press. - 45 - Appendix A Personal Reflection When It came to choosing a project, this one immediately stuck out to me. Board games and tabletop gaming is a big passion of mine, and when I think of the word “Tabletop”, games are the first thing that come to my mind. My ambition in my career once I graduate is to become a games developer, and while there were no projects proposed which directly translated to games development programming, I saw an opportunity here to add an element of “games” to a project which already had an exciting base in that it involved robotics and, especially, a Baxter robot. After all, robots are cool, exciting and wonderful. Even now after dozens of hours working on Lucas, the sci-fi geek in me still revels in the fact that I have made a real life robot play a board game with a human13. Such was my excitement that I went a little overboard with my original concept, and my ambition was rightfully tempered by my supervisors early on so as to save me from setting goals which would have been impossible to achieve in the time given. Upon entering this project, I was venturing into a completely unknown realm of computer science: Robotics. This was the first time I had actually worked on a robot since I had access to a LEGO Mindstorms kit when I went to LEGOLand as a child. Similarly I had no previous experience of actually programming an AI algorithm, the second year Artificial Intelligence module I took unfortunately suffered from some complications, and there was only a small practical coursework in the end. My experience programming computer vision was also extremely limited, as we had had only one specific coursework assigned to us during the second year Computer Vision module. When I started working on this project, I perceived it to be immensely daunting, a task of almost Herculean scale, comparable to slaying a hydra. I have an unfortunate habit of shying away from work that seems impossible, fearing that I would be defeated, proved right and driven further away from the task at hand. As such, I spent a lot of time designing and planning solutions for each aspect of the project without actually working on the implementation. Now, having successfully gone through and implemented the project, having actually made a program that makes Lucas play a game with me, I am ecstatic that I managed to achieve the final program that I did, relieved that I managed to do it within the timeframe, but mostly, I am frustrated at myself for putting myself off of it for so long. I found that once I started working 13 Yes, even when I was encountering those frustrating bugs that make no sense and take hours to figure out, before turning out to be a stupid mistake such as a misplaced set of parentheses. - 46 with the robot itself, it was much clearer than I had anticipated. I feel that if I hadn’t scared myself off of working on Lucas and exploring the capacity of what it can do, I could have delivered a much more refined and advanced system incorporating some of the features I have detailed in the future work section of this report. The vision systems aren’t the 400-line goliaths I had envisioned. The mechanics are precise and were the easiest part to implement. The whole program fits into about 500 lines of code if you remove the variable definitions. Lucas performs its actions smoothly and efficiently. RethinkRobotics have made a robot that is an absolute pleasure to work with. In positioning the arms to gather joint angle values I could feel when the arm was reaching the limit on one of the 7 motors in the arm and adjust accordingly, the arm moving smoothly and in accord with whatever way I pushed it. At times, Lucas can feel strangely “alive” such as when the arms are in danger of colliding, it will reflexively stop and move one out of the way. Despite my slow start, I am pleased with the outcome achieved, although I am annoyed at the fact that there were a few bugs in the version of the code which I used to gather data and record the demonstration video. Hearing the subject matter of the final projects of some of my course-mates, I am very glad to have chosen such a unique and exciting project. I am still extremely surprised that myself and Ben were the only two students to apply to work with a Baxter robot for a few months. Due to the nature of this project, all of the practical development of the implementation had to be done in the labs, as I couldn’t bring Lucas home to run tests on. I feel this helped me to focus on my work once I started working as, when I was in the lab, I was not exposed to the distractions that are legion at my desk at home. One thing I very much appreciated about working on this project was the environment in the robotics lab, as well as the resources provided to me so that I could always have access to what I needed for this project. I had access to the lab 24/7, and even had the cost of the materials for the game pieces reimbursed. At first I was concerned about working in the lab with a robot that is a shared resource, but after a while I became comfortable in the lab and it was made clear that the other people working there were more than willing to give up time working with Lucas whenever I needed it, and were also willing to offer advice when I was stuck with a bug or needed to get Lucas to do something specific through the Baxter API. It reminded me of the internship I had last summer, where I was programming factory operator interfaces, and also had zero experience in the subject matter prior to turning up and getting stuck in. At first it seems almost overwhelming, working in the same office as people who have been doing this for years and must be sick of untrained newcomers to the field asking for advice constantly, but after a while the realisation dawns that the people working around you are human too, and were once in the same position. - 47 - Overall, I am glad that I chose this project. It was exciting, and appealed greatly to the side of me that loves the fantastical and futuristic, it pushed me out of my comfort zone and into an area of computing I had no opportunity to explore before in an academic context, one which has been romanticised, explored and parodied in popular science-fiction and forms one of the core aspects of some of my favourite science fiction franchises14. The result of achieving the final project implementation is extremely satisfying in comparison to a lot of computer science projects as it has a real, visible, tangible effect on the physical world as Lucas moves blocks around a board in front of you. I have learnt a lot of useful programming skills in areas of computer science I hold a great interest in, I have had the opportunity to work with a fantastic, highly sophisticated modern robot and I am satisfied with the final result of my project. I do not think I could ask for a better end to my 5-year journey to achieving a BSc15. 14 One example of a character who programs a robot that can play board games? Darth Vader. Anakin Skywalker created C3-P0, who plays Holochess against Chewbacca in Star Wars Episode IV: A New Hope, where the famous line “Let the Wookiee win” was uttered. 15I spent 2 years in York studying Astrophysics, which, as it turns out, is quite difficult. During this time, I made some excellent friends and met my partner, while living in one of the most wonderful cities in Britain. As such I do not regret one moment of it. - 48 - Appendix B Ethical Issues Addressed B. i: Ethical issues I do not think there are any ethical issues to be raised for this project as, although it pits a robot in direct opposition to a human, it is not with intent to cause harm or allow a human to come to harm. - 49 - Appendix C Record of External Materials Used Few external materials were used in this project. The only source from which code was taken directly, in order to figure out how OpenCV gets and draws contours (although the code which was taken has since been heavily modified and dissected in this project) was: http://opencvpython.blogspot.co.uk/2012/06/hi-this-article-is-tutorial-which-try.html This source is a tutorial on finding contours in images using OpenCV and was accessed in late April 2015. - 50 - Appendix D Full Rules of the Game D. i: Components One board, plywood, marked in pencil. 20 game pieces, wood, marked out with tape. D. ii: The Tiles Each game piece or tile shows two possible paths across a game board space, depicted in white on a black background. These two paths connect the centre of each side to the centre of another side of the tile face. Paths on a tile may cross each other, but the path continues along the single most straightforward route across the tile, uninterrupted and independently of the others. D. iii: Setup Place the board on a table. If playing with a Baxter robot, place the board in the exact configuration required relative to the Baxter robot. This can be determined by moving the left hand to the PickUpD joint angles found in coords.py and aligning the pickup space of the board underneath it. Place the 20 game tiles in a face-down pile on the table. D. iv: Playing the game At the start of each turn, the current player draws a tile at random. Players take turns placing tiles on the board. Each player must start along a board edge. Once you have placed your first tile, your path starts at the board edge and follows the path on the tile. A player may not willingly direct their path off of the game board unless they have no other option. A player may place their tile in any orientation so long as it lines up with the grid spaces on the board. A player is eliminated from the game if the open end of their path connects to an edge of the game board. - 51 - If their path collides with another tile, the path follows the path on that tile and any subsequent tiles until the path reaches an empty space or a board edge. D. v: Winning the Game A player becomes the winner when they are the only player remaining who has not been eliminated. D. vi: FAQ Q: What happens if the two paths meet? A: The game ends in a draw, as both paths now lead off the edge. Congratulations, you have both lost. Q: How do I keep track of which path belongs to which player? A: Use your mind. Alternatively, use a small white token to mark the progress of your path. Q: On a type 1 tile (straight across), can I use any of the paths off of the tile as they all connect in the middle? A: No, on the type 1 tile you can only use the path leading across the tile. - 52 - Appendix E Project Code and Other Materials of Interest E. i: coords.py coords.py has been excluded from this Appendix on the grounds that it is roughly 12 A4 pages of joint angle values for Lucas. If you would like to inspect coords.py, feel free to access it at https://github.com/sc12pas/TTR/blob/master/src/coords.py E. ii: vision.py #vision.py #Full code for the Final Year Project: #"Tabletop Robotics: Board Games With a Baxter Robot" # by Pascal Alexander Siddons, [email protected] #----------------Importing required libraries-------------------------------# import numpy as np import rospy import cv2 import cv_bridge import time import sys import baxter_interface as bax from select import select import coords #The file containing all of the necessary joint angles from sensor_msgs.msg import Image rospy.init_node('Game_run') #-----------------Defining object classes-----------------------------------# class Point: """A class to describe objects representing points in 2D coordinate systems""" def __init__(self): self.x = 0 self.y = 0 class BoardSpace: """A class to describe objects representing the 20 spaces on the game board""" def __init__(self,posU,posD,piece,x1,y1,x2,y2): self.posU = posU - 53 self.posD = posD self.piece = piece self.x1 = x1 self.y1 = y1 self.x2 = x2 self.y2 = y2 #---------------Defining global variables and the parameters for the robots arms,-# #------------------------Gripper and cameras.-------------------------------------# limbR = bax.Limb('right') limbL = bax.Limb('left') gripper = bax.Gripper('left') left_camera = bax.CameraController('left_hand_camera') left_camera.resolution = (640,400) left_camera.open() left_image = None right_camera = bax.CameraController('right_hand_camera') right_camera.resolution = (640,400) right_camera.open() right_image = None board_spaces =  current_space = 0 next_space = 0 current_piece = 0 direction = "" gameLoss = False #------------------------Defining all of the board spaces-------------------# h1=195 h2=260 h3=335 h4=400 h5=470 h6=535 v1=44 v2=113 v3=174 v4=250 v5=315 #The first two parameters correspond to the joint angles required to #navigate to that particular space A1 = BoardSpace(coords.A1U, coords.A1D, 0, h1,v1,h2,v2) board_spaces.append(A1) #The third is the piece currently occupying that space #While the last four describe the area this space occupies in the orthogonal view A2 = BoardSpace(coords.A2U, coords.A2D, 0, h2,v1,h3,v2) board_spaces.append(A2) A3 = BoardSpace(coords.A3U, coords.A3D, 0, h3,v1,h4,v2) - 54 board_spaces.append(A3) A4 = BoardSpace(coords.A4U, coords.A4D, 0, h4,v1,h5,v2) board_spaces.append(A4) A5 = BoardSpace(coords.A5U, coords.A5D, 0, h5,v1,h6,v2) board_spaces.append(A5) B1 = BoardSpace(coords.B1U, coords.B1D, 0, h1,v2,h2,v3) board_spaces.append(B1) B2 = BoardSpace(coords.B2U, coords.B2D, 0, h2,v2,h3,v3) board_spaces.append(B2) B3 = BoardSpace(coords.B3U, coords.B3D, 0, h3,v2,h4,v3) board_spaces.append(B3) B4 = BoardSpace(coords.B4U, coords.B4D, 0, h4,v2,h5,v3) board_spaces.append(B4) B5 = BoardSpace(coords.B5U, coords.B5D, 0, h5,v2,h6,v3) board_spaces.append(B5) C1 = BoardSpace(coords.C1U, coords.C1D, 0, h1,v3,h2,v4) board_spaces.append(C1) C2 = BoardSpace(coords.C2U, coords.C2D, 0, h2,v3,h3,v4) board_spaces.append(C2) C3 = BoardSpace(coords.C3U, coords.C3D, 0, h3,v3,h4,v4) board_spaces.append(C3) C4 = BoardSpace(coords.C4U, coords.C4D, 0, h4,v3,h5,v4) board_spaces.append(C4) C5 = BoardSpace(coords.C5U, coords.C5D, 0, h5,v3,h6,v4) board_spaces.append(C5) D1 = BoardSpace(coords.D1U, coords.D1D, 0, h1,v4,h2,v5) board_spaces.append(D1) D2 = BoardSpace(coords.D2U, coords.D2D, 0, h2,v4,h3,v5) board_spaces.append(D2) D3 = BoardSpace(coords.D3U, coords.D3D, 0, h3,v4,h4,v5) board_spaces.append(D3) D4 = BoardSpace(coords.D4U, coords.D4D, 0, h4,v4,h5,v5) board_spaces.append(D4) D5 = BoardSpace(coords.D5U, coords.D5D, 0, h5,v4,h6,v5) board_spaces.append(D5) #---------------------------Code to retrieve images from Lucas------------------# - 55 - def get_right(msg): global right_image right_image desired_encoding='rgb8') = cv_bridge.CvBridge().imgmsg_to_cv2(msg, def get_left(msg): global left_image left_image = cv_bridge.CvBridge().imgmsg_to_cv2(msg, desired_encoding='rgb8') left_sub = rospy.Subscriber( 'cameras/left_hand_camera/image', Image, get_left ) right_sub = rospy.Subscriber( 'cameras/right_hand_camera/image', Image, get_right ) #--------------------------Functions for everything needed to play the game-----# def place(num): """A function to instruct Lucas to place a block in a particular space""" global current_piece global current_space global board_spaces space = board_spaces[num] limbL.move_to_joint_positions(coords.ResetPos) limbL.move_to_joint_positions(space.posU) limbL.move_to_joint_positions(space.posD) gripper.open() space.piece = current_piece current_space = num limbL.move_to_joint_positions(space.posU) limbL.move_to_joint_positions(coords.ResetPos) def startTurn(): """A function instructing Lucas to wait for the start of its turn before scanning the board to see where the Human has placed their tile""" global current_piece current_piece = 0 limbL.move_to_joint_positions(coords.ResetPos) limbL.move_to_joint_positions(coords.PickU) limbL.move_to_joint_positions(coords.PickD) i = 0 while current_piece == 0: current_piece = IDBlock(left_image) rospy.sleep(5) i+=1 if i >= 6:#After 30 seconds of inaction by the human player, Lucas assumes victory print "Huzzah! Another glorious victory for your robotic overlord!" sys.exit() limbL.move_to_joint_positions(coords.PickU) limbL.move_to_joint_positions(coords.ResetPos) scanBoard() limbL.move_to_joint_positions(coords.ResetPos) - 56 def move_to(bs): #A function instructing Lucas to move the left arm to a particular space limbL.move_to_joint_positions(coords.ResetPos) limbL.move_to_joint_positions(bs.posU) limbL.move_to_joint_positions(bs.posD) def move_from(bs): #A function instructing Lucas to move back from a particular space limbL.move_to_joint_positions(bs.posU) limbL.move_to_joint_positions(coords.ResetPos) def pickUp(): #A Function instructing Lucas to pick up its tile for the turn limbL.move_to_joint_positions(coords.PickU) limbL.move_to_joint_positions(coords.PickD) current_piece = IDBlock(left_image) gripper.close() rospy.sleep(2) limbL.move_to_joint_positions(coords.PickU) limbL.move_to_joint_positions(coords.ResetPos) def rotate(): # A function instructing Lucas to rotate its piece for the turn by 90 degrees limbL.move_to_joint_positions(coords.PickU) limbL.move_to_joint_positions(coords.rotateU) limbL.move_to_joint_positions(coords.rotateD) gripper.open() rospy.sleep(2) limbL.move_to_joint_positions(coords.rotateU) pickUp() def nextSpace(piece): """A function calculating, given the current piece and with knowledge of the current space and direction, what the space Lucas will be placing a piece in in its next turn will be""" global current_space global current_piece global next_space global direction if direction == "up": print "piece = ", piece if piece == 1: next_space = current_space - 5 if next_space <= -1: next_space = 66 elif piece == 2: if current_space == (4 or 9 or 14): next_space = 66 else: next_space = current_space + 1 elif piece == 3: if current_space == (0 or 5 or 10): next_space = 66 - 57 else: next_space = current_space - 1 elif direction == "down": if piece == 1: next_space = current_space + 5 if next_space >= 20: next_space = 66 elif piece == 2: if current_space == (5 or 10 or 15): next_space = 66 else: next_space = current_space - 1 elif piece == 3: if current_space == (9 or 14 or 19): next_space = 66 else: next_space = current_space + 1 elif direction == "right": if piece == 1: if current_space == 4 or current_space ==9 or current_space ==14 or current_space ==19: next_space = 66 else: next_space = current_space + 1 elif piece == 2: if 1 <= current_space <= 4: next_space = 66 else: next_space = current_space - 5 elif piece == 3: if 16 <= current_space <= 19: next_space = 66 else: next_space = current_space + 5 elif direction == "left": if piece == 1: if current_space == (0) or current_space == 5 or current_space == 10 or current_space == (15): next_space = 66 else: next_space = current_space - 1 elif piece == 2: next_space = current_space + 5 if next_space >= 20: next_space = 66 elif piece == 3: next_space = current_space - 5 if next_space <= -1: next_space = 66 else: print "+++?????+++ Out of Cheese Error. Redo From Start" sys.exit() - 58 if next_space <= -1: next_space = 66 elif next_space >= 20: next_space = 66 #Defining next_space as 66 tells Lucas the next space is off of the board and that it has lost return next_space def changeDirection(): """A function instructing Lucas to update the direction it is moving around the board in given the type of piece it has just placed""" global current_space global current_piece global next_space global direction if direction == "up": if current_piece == 3: direction = "left" elif current_piece == 2: direction = "right" elif direction == "down": if current_piece == 3: direction = "right" elif current_piece == 2: direction = "left" elif direction == "right": if current_piece == 3: direction = "down" elif current_piece == 2: direction = "up" elif direction == "left": if current_piece == 3: direction = "up" elif current_piece == 2: direction = "down" def scanBoard(): """A function which scans all of the empty spaces on the board to see if any have changed. The one which is most likely to have changed is then scanned by Lucas""" global board_spaces limbL.move_to_joint_positions(coords.ResetPos) limbL.move_to_joint_positions(coords.left_back) limbR.move_to_joint_positions(coords.ortho_view) board = right_image thresh, bd = cv2.threshold(board, 127,255,cv2.THRESH_BINARY) array =  - 59 maxspace = -1 meanmax = 0 for i in range(len(board_spaces)): if board_spaces[i].piece == 0: bs = board_spaces[i] space = bd[bs.y1:bs.y2, bs.x1:bs.x2] if np.mean(space) >= 0.4: if np.mean(space) >= meanmax: meanmax = np.mean(space) maxspace = i line = cv2.line(bd, (bs.x1,bs.y1), (bs.x2,bs.y2), (0,255,0), 1) if maxspace >= 0: bs = board_spaces[maxspace] cv2.imwrite('conts.jpg',bd) limbR.move_to_joint_positions(coords.tuck_right) move_to(bs) bs.piece = IDBlock(left_image) move_from(bs) limbR.move_to_joint_positions(coords.tuck_right) def turnOne():# A function instructing Lucas with what to do on its first turn. global board_spaces global direction global next_space gripper.calibrate() limbR.move_to_joint_positions(coords.tuck_right) limbL.move_to_joint_positions(coords.ResetPos) startTurn() pickUp() if board_spaces.piece == 0: place(17) direction = "up" next_space = 12 else: place(12) direction = "down" next_space = 7 def IDBlock(img):# A function to Identify the type of a block img = img[0:300, 160:520] thresh, im = cv2.threshold(img, 127,255,cv2.THRESH_BINARY) imgray = cv2.cvtColor(im,cv2.COLOR_BGR2GRAY) ret,thresh = cv2.threshold(imgray,127,255,0) contours, hierarchy cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE) cv2.drawContours(im,contours,-1,(0,255,0),-1) my_contours =  tile_type = 0 for cnt in contours: if cv2.contourArea(cnt)>2000: my_contours.append(cnt) cv2.drawContours(im,my_contours,-1,(255,0,0),-1) if(len(my_contours)==1): tile_type = 1 elif(len(my_contours)==0): = - 60 tile_type = 0 else: centres =  for i in range(len(my_contours)): mom = cv2.moments(my_contours[i]) centre = Point() centre.x = (int(mom['m10']/mom['m00'])) centre.y = (int(mom['m01']/mom['m00'])) centres.append(centre) line = cv2.line(im, (centres.x,centres.y), (centres.x,centres.y), (0,255,0), 1) if (centres.x > centres.x) and (centres.y > centres.y): tile_type = 2 elif (centres.x > centres.x) and (centres.y > centres.y): tile_type = 2 else: tile_type = 3 print tile_type return tile_type def boardStatePrint(): #This just prints out a nice little display of Lucas' current perspective of the board state into the console print " 1 2 3 4 5 " print "A","|",board_spaces.piece,"|",board_spaces.piece,"|",board_spaces.piece," |",board_spaces.piece,"|",board_spaces.piece,"|" print "B","|",board_spaces.piece,"|",board_spaces.piece,"|",board_spaces.piece," |",board_spaces.piece,"|",board_spaces.piece,"|" print "C","|",board_spaces.piece,"|",board_spaces.piece,"|",board_spaces.piec e,"|",board_spaces.piece,"|",board_spaces.piece,"|" print "D","|",board_spaces.piece,"|",board_spaces.piece,"|",board_spaces.piec e,"|",board_spaces.piece,"|",board_spaces.piece,"|" while left_image==None: pass #-------------Calling turnOne() and the main game loop-------------------------------------# turnOne() #Start the game! while gameLoss == False: # The main game loop, where the action happens and it all comes together. startTurn() print "State: ", current_piece, " ", direction, " ", current_space, board_spaces[current_space].piece current_space = next_space if board_spaces[current_space].piece != 0: while board_spaces[current_space].piece != 0: current_space = nextSpace(board_spaces[current_space].piece) changeDirection() if current_space == 66: gameLoss = True break - 61 print "State: ", current_piece, current_space, board_spaces[current_space].piece " ", direction, " ", pickUp() print "State: ", current_piece, " ", direction, " ", current_space if current_piece == 1: place(current_space) next = nextSpace(current_piece) if next == 66: gameLoss = True elif current_piece == 2: print "State: ", current_piece, " ", direction, " ", current_space next = nextSpace(current_piece) altnext = nextSpace(3) print "next: ", next, " , ", altnext next == 13: if next == 66: if altnext == 66: place(current_space) gameLoss = True else: rotate() place(current_space) elif next == 6 or next == 7 or next == 8 or next == 11 or next == 12 or place(current_space) next_space = next elif altnext == 6 or altnext == 7 or altnext == 8 or altnext == 11 or altnext == 12 or altnext == 13: rotate() place(current_space) next_space = altnext else: place(current_space) next_space = next elif current_piece == 3: next = nextSpace(current_piece) altnext = nextSpace(2) print "next: ", next, " , ", altnext if next == 66: if altnext == 66: place(current_space) gameLoss = True else: rotate() place(current_space) elif next == 6 or next ==7 or next ==8 or next ==11 or next ==12 or next == 13: place(current_space) next_space = next elif altnext == 6 or altnext ==7 or altnext ==8 or altnext ==11 or altnext ==12 or altnext ==13: rotate() place(current_space) next_space = altnext - 62 else: place(current_space) next_space = next changeDirection() boardStatePrint() print next_space if gameLoss == True: print "+++Divide By Cucumber Error. Please Reinstall Universe And Reboot+++" sys.exit() E. iii: Demonstration Video The (less-than-perfect) demonstration video16 for the final implementation of this project can be found on YouTube here: https://www.youtube.com/watch?v=YEtFgFRH7wk Alternatively you can download the non-annotated version from DropBox here: https://www.dropbox.com/s/nbhjdbjevfaxi47/Final%2520Project%2520Demo%2520Tabletop%2520Robotics.mp4?dl=0 E. iv: GitHub Repository The GitHub repository containing all of the files for this project can be found here: https://github.com/sc12pas/TTR 16 I went into the labs on 12/5/15 to attempt to re-record the demo video but found that Lucas had been moved and I could not recreate the lighting environment necessary while it was in its new position, and struggled to place the board at the right height for the mechanical system to operate fully.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project