Skeletal animation for games in Haskell

I recently revisited my project to bring concepts from Jason Gregory's Game Engine Architecture into Haskell and found that they are translating quite well. The relational ideas I've been playing around with mesh nicely though there are still some rough edges. This is not a highly efficient implementation but I think that the code allows for some nice possible optimizations.

I decided that I wanted to implement skeletal animation since I figured that even a great set of free game sprites would only go so far. Fortunately GEA has a great description of this with a precise set of terminology, and helped me get this implementation going. I apologize ahead of time for any butchering of terminology that will likely happen.

As most of you know, skeletal animation binds meshes to a system of joints, so that manipulating the joints will deform the meshes. This allows you to get your characters into different poses without drawing them from scratch, and later, poses can also be blended to get combinations of movements, like running and shooting. I will skip most of the math specifics since there are a lot of resources that go into the details.

So far my code just gets from this:

To this:

But eventually this should let me get a lot of movement out of the characters from a small set of base art.

I'm still using a similar setup to Experimenting with Game Concepts in Haskell. So an entity like a player is split up into several components, and they are only associated with each other by a shared id. After using this for a while I feel that the single shared id is too limiting and a more standard relational design would work better, but the single id is a good starting point.

As before the entire game state is stored in a single value that is updated in a purely functional way. The game state value is stored in a transactional variable (TVar) that is updated on each game step.

data GameState = GameState { player :: HaskQuery.Relation (Component Player) (HaskQuery.IntIndex.IntIndex),
                             position :: HaskQuery.Relation (Component Position) (HaskQuery.IntIndex.IntIndex),
                             texture :: HaskQuery.Relation (Component Texture) (HaskQuery.IntIndex.IntIndex),
                             renderer :: HaskQuery.Relation (Component Renderer) (HaskQuery.IntIndex.IntIndex),
                             axisAlignedBoundingBox :: HaskQuery.Relation (Component AxisAlignedBoundingBox) (HaskQuery.IntIndex.IntIndex),
                             skeleton :: HaskQuery.Relation (Component Skeleton) (HaskQuery.IntIndex.IntIndex),
                             flatSkeleton :: HaskQuery.Relation (Component (FlatSkeleton Bone)) (HaskQuery.IntIndex.IntIndex),
                             flatSkeletonJoint :: HaskQuery.Relation (Component (FlatSkeleton Joint)) (HaskQuery.IntIndex.IntIndex),
                             _globalPose :: HaskQuery.Relation (Component GlobalSkeletonPose) (HaskQuery.IntIndex.IntIndex),
                             _mesh :: HaskQuery.Relation (Component Mesh) (HaskQuery.IntIndex.IntIndex),
                             _meshSkeletonBinding :: HaskQuery.Relation (Component MeshSkeletonBinding) (HaskQuery.IntIndex.IntIndex),
                             _animationClip :: HaskQuery.Relation (Component AnimationClip) (HaskQuery.IntIndex.IntIndex),
                             _animationState :: HaskQuery.Relation (Component AnimationState) (HaskQuery.IntIndex.IntIndex),
                             currentId :: GameEntityIdentifier
                           }

The key fields that are used for posing are:

 
position :: HaskQuery.Relation (Component Position) (HaskQuery.IntIndex.IntIndex),
flatSkeletonJoint :: HaskQuery.Relation (Component (FlatSkeleton Joint)) (HaskQuery.IntIndex.IntIndex),
_globalPose :: HaskQuery.Relation (Component GlobalSkeletonPose) (HaskQuery.IntIndex.IntIndex),
_mesh :: HaskQuery.Relation (Component Mesh) (HaskQuery.IntIndex.IntIndex),
_meshSkeletonBinding :: HaskQuery.Relation (Component MeshSkeletonBinding) (HaskQuery.IntIndex.IntIndex),

In this setup a skeleton is a tree of joints. Joints are affine transformations so linear transformation + translation or a 4x3 matrix. A lot of game engines don't store the full 4x3 matrix and store the joint as a Scale, Rotation (Quaternion) and Translation. I used the excellent Linear library for a lot of data structures and math functions.

So we take the base joints, add a notion of a primary key and parent id, then collect them in a relation using the relation library I've been experimenting with in Building Gtk2HS guis with queries.

data Joint = Joint { 
    _transform :: Linear.V3 (OpenGL.GLdouble), --This will need to be renamed later
    _scale :: (OpenGL.GLdouble), 
    _quaternion :: Linear.Quaternion (OpenGL.GLdouble)}
             deriving (Show)

data ComponentWithParent a = BoneWithParent { _parentId :: Int, _boneId :: Int, _bone :: a}
                             deriving (Show)


data FlatSkeleton a = FlatSkeleton { _rootId :: Int, _bones :: HaskQuery.Relation (ComponentWithParent a) (HaskQuery.IntIndex.IntIndex) }
                      deriving (Show)

Here's the skeleton that we're using, which is shown by the yellow lines in the images at the start.

let originalSkeleton = TestGame.Skeleton.flatSkeleton (-1) [
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = -1, TestGame.Types._boneId =0, TestGame.Types._bone = TestGame.Skeleton.defaultJoint {TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (0.00)} },
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = 0, TestGame.Types._boneId =1, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._transform = Linear.V3 100 0 0, TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) 0 }},
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = 1, TestGame.Types._boneId =2, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._transform = Linear.V3 100 0 0,  TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (0) }},
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = -1, TestGame.Types._boneId =3, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (-pi)} },
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = 3, TestGame.Types._boneId =4, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._transform = Linear.V3 100 0 0, TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (0.0) }},
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = 4, TestGame.Types._boneId =5, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._transform = Linear.V3 100 0 0,  TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (0.0) }},
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = -1, TestGame.Types._boneId =6, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (-0.5*pi)} },
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = 6, TestGame.Types._boneId = 7, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._transform = Linear.V3 100 0 0, TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (-0.25*pi)} },
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = 6, TestGame.Types._boneId =8, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._transform = Linear.V3 100 0 0, TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (0.25*pi)} },
    TestGame.Types.BoneWithParent {TestGame.Types._parentId = -1, TestGame.Types._boneId =9, TestGame.Types._bone = (TestGame.Skeleton.defaultJoint) {TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (0.50*pi)} }
]

The skeleton then has to be attached to meshes, in this image is a mesh for each arm, then one for the neck/head. The skeleton has an initial pose called the bind pose that is fixed when the meshes are attached. The mesh vertices will be transformed from where they are relative to the bind pose to where they are relative to the current pose.

Here is one of the purple arm meshes, which is a triangle strip in entityId order. I will omit the other meshes for the remaining discussion.

TestGame.Types.MeshComponent $ TestGame.Mesh.mesh $ \ meshRelation ->
    HaskQuery.insertRows meshRelation [  TestGame.Types.Component { TestGame.Types._entityId = 1, TestGame.Types._component = TestGame.Mesh.vertex3 0 25 0 },
        TestGame.Types.Component { TestGame.Types._entityId = 2, TestGame.Types._component = TestGame.Mesh.vertex3 0 (-25) 0},
        TestGame.Types.Component { TestGame.Types._entityId = 3, TestGame.Types._component = TestGame.Mesh.vertex3 100 (25) 0},
        TestGame.Types.Component { TestGame.Types._entityId = 4, TestGame.Types._component = TestGame.Mesh.vertex3 100 (-25) 0},
        TestGame.Types.Component { TestGame.Types._entityId = 5, TestGame.Types._component = TestGame.Mesh.vertex3 200 (25) 0},
        TestGame.Types.Component { TestGame.Types._entityId = 6, TestGame.Types._component = TestGame.Mesh.vertex3 200 (-25) 0},
        TestGame.Types.Component { TestGame.Types._entityId = 7, TestGame.Types._component = TestGame.Mesh.vertex3 300 (0) 0}
    ]

Now we need to attach this mesh to the original bind pose. The bind pose will always be used to determine the initial relationship between a joint and a vertex. This relation is a mapping from vertices to joints with a weight attached. So the vertices around the elbow will be jointly determined by the two parts of the arm joint. This is a natural fit for a relational model, though I am using a lot of manual id manipulation which I hope to reduce.

TestGame.Types.MeshSkeletonComponent $ TestGame.Mesh.meshSkeleton $ \ meshSkeletonBinding ->
    HaskQuery.insertRows meshSkeletonBinding [
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 1, TestGame.Types._jointId=0, TestGame.Types._bindWeight=1},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 2, TestGame.Types._jointId=0, TestGame.Types._bindWeight=1},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 3, TestGame.Types._jointId=0, TestGame.Types._bindWeight=0.5},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 3, TestGame.Types._jointId=1, TestGame.Types._bindWeight=0.5},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 4, TestGame.Types._jointId=0, TestGame.Types._bindWeight=0.5},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 4, TestGame.Types._jointId=1, TestGame.Types._bindWeight=0.5},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 5, TestGame.Types._jointId=1, TestGame.Types._bindWeight=1},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 6, TestGame.Types._jointId=1, TestGame.Types._bindWeight=1},
        TestGame.Types.VertexJointBinding { TestGame.Types._vertexId = 7, TestGame.Types._jointId=2, TestGame.Types._bindWeight=1}]
                                                                 

Now we have our skeleton in bind pose bound to the meshes. Now we want to change the skeleton pose and transform the mesh appropriately.

Let's create the new pose with some help from the lens library:

let pose = originalSkeleton 
    & (TestGame.Types.bones %~ (\ uBones -> HaskQuery.update uBones (( == 0) . TestGame.Types._boneId  ) 
        (TestGame.Types.bone %~ (\bone -> bone { TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (-0.25) } ))))
    & (TestGame.Types.bones %~ (\ uBones -> HaskQuery.update uBones (( == 1) . TestGame.Types._boneId  ) 
        (TestGame.Types.bone %~ (\bone -> bone { TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (1) }  ))))
    & (TestGame.Types.bones %~ (\ uBones -> HaskQuery.update uBones (( == 2) . TestGame.Types._boneId  ) 
        (TestGame.Types.bone %~ (\bone -> bone { TestGame.Types._quaternion = Linear.axisAngle (Linear.V3 0 0 1) (0.75) }  ))))   

So now for the math, which I'll only cover at a high level. The skeleton in bind pose is a tree of transformations, each one relative to its parent. We need to turn this into a set of transformations, each one relative to the root coordinate system, instead of its parent.

To do this we traverse the tree, multiplying the transformations as we go down and collecting all the results. This is a recursive query on the bind pose relation.

flatJointSkeletonToGlobalPose :: FlatSkeleton Joint -> GlobalSkeletonPose
flatJointSkeletonToGlobalPose flatSkeletonJointPose =
    let emptyGlobalSkeletonPose = HaskQuery.reindex HaskQuery.empty (HaskQuery.IntIndex.intIndex _boneId)
    in  HaskQuery.insertInto emptyGlobalSkeletonPose $ do
            rootJoint <- HaskQuery.selectWithIndex (HaskQuery.IntIndex.intIndexSelector) (_bones flatSkeletonJointPose) (_rootId flatSkeletonJointPose)
            selectGlobalPoses (rootJoint { _bone = jointToMatrix $ (_bone rootJoint)}) flatSkeletonJointPose

Here we multiply the current local joint by the accumulated global joint.

selectGlobalPoses :: GlobalJointPose -> FlatSkeleton Joint -> HaskQuery.Cont (b -> b) GlobalJointPose
selectGlobalPoses joint flatSkeleton = do
    let parentGlobalJointPoseMatrix = (_bone joint)
    produceAndRecurse joint $ do
        b <- HaskQuery.selectWithIndex (HaskQuery.IntIndex.intIndexSelector) (_bones flatSkeleton) (_boneId joint)
        let globalJoint = b { _bone = parentGlobalJointPoseMatrix Linear.!*! (jointToMatrix $ (_bone b))}
        selectGlobalPoses globalJoint flatSkeleton

When skinning, we are going to take the global pose of the bind pose, invert it, apply the inverse to the meshes, then apply the current pose to the meshes. If we think of a hand mesh vertex at (1,1,1) from a joint, assuming no scaling, then after inverting, the vertex will be (1,1,1). Then we need to move that hand mesh vertex to where the joint currently is, so we apply the current pose global transformation which should be (1,1,1) from the current joint. Please check GEA or other resources for a better explanation.

The GameState value allows multiple renderers to be associated with an entity. There is a mesh renderer to render the skinned mesh in purple, and another one will render the skeleton in yellow. The mesh renderer will find the bind pose flatSkeletonJointComponent, mesh meshComponent and current pose globalPose with queries.

For each mesh it will start a triangle strip, and for each vertex in the mesh it will find all joint bindings and apply the inverse global bind pose and current global pose to the vertex with appropriate weights. This code was originally written with the single mesh case, now that there are multiple meshes, reordering the queries and only calculating the inverse once would probably be a good idea.

flatSkeletonMeshGlobalPoseRenderer :: (OpenGL.ColorComponent a) => OpenGL.Color4 a -> Renderer
flatSkeletonMeshGlobalPoseRenderer color entityId gameState  = do
    OpenGL.color $ color
    _ <- HaskQuery.runQueryM $ do
        Component { _component = positionComponent } <- TestGame.GameState.selectEntityByIdM gameState position entityId
        Component { _entityId = _meshEntityId, _component = meshComponent} <- TestGame.GameState.selectEntityByIdM gameState _mesh entityId
        Component { _entityId = _meshEntityId, _component = flatSkeletonJointComponent} <- TestGame.GameState.selectEntityByIdM gameState flatSkeletonJoint entityId
        Component { _entityId = _meshEntityId, _component = globalPose} <- TestGame.GameState.selectEntityByIdM gameState _globalPose entityId
        HaskQuery.executeM $ do
            let skinningMatrices = TestGame.Skeleton.flatJoinSkeletonToInverseGlobalPose flatSkeletonJointComponent
            OpenGL.preservingMatrix $ do
                OpenGL.translate (OpenGL.Vector3 (fromIntegral (xPosition positionComponent))
                                                 (fromIntegral (yPosition positionComponent))
                                                 (0 :: OpenGL.GLdouble))
                OpenGL.renderPrimitive OpenGL.TriangleStrip $ do
                    _ <- HaskQuery.runQueryM $ do
                        Component { _entityId = vertexId, _component = vertexComponent} <- HaskQuery.selectM (_meshVertices meshComponent)
                        Component { _component = skeletonBinding } <- TestGame.GameState.selectEntityByIdM gameState _meshSkeletonBinding entityId
                        HaskQuery.executeM $ do
                            let collector :: (Linear.V4 OpenGL.GLdouble, (Linear.V4 OpenGL.GLdouble -> Linear.V4 OpenGL.GLdouble -> Linear.V4 OpenGL.GLdouble))
                                collector = ((Linear.V4 0 0 0 1), (\ vector result -> vector Linear.^+^ result))
                            let vertex = HaskQuery.runQueryWithCollector collector $ do
                                    VertexJointBinding {_jointId = jointId, _bindWeight = bindWeight} <- HaskQuery.selectWithIndex HaskQuery.IntIndex.intIndexSelector (_meshVertexBinding skeletonBinding) vertexId
                                    BoneWithParent { _bone = skinningMatrix} <- HaskQuery.selectWithIndex HaskQuery.IntIndex.intIndexSelector  skinningMatrices jointId
                                    BoneWithParent { _bone = globalPoseMatrix } <- HaskQuery.selectWithIndex HaskQuery.IntIndex.intIndexSelector  globalPose jointId
                                    let OpenGL.Vertex3 x y z = vertexComponent
                                    return $  bindWeight Linear.*^ ((globalPoseMatrix Linear.!*! skinningMatrix) Linear.!* (Linear.V4 x y z 1))
                            let Linear.V4 px py pz _ = vertex
                            OpenGL.vertex $ OpenGL.Vertex3 px py pz
                        return ()
                    return ()
    return ()

While working on this I found a lot of advantages to representing data relationally. A recursive query gave a nice way of calculating global poses. VertexJointBinding which attaches meshes to bind pose skeletons was stored separately and did not change the definition of skeletons or meshes.

This code probably has a decent bit of overhead from using IntMaps under the hood everywhere, but the above queries could be used to build efficient Vectors to store static data. Data can be stored in a flexible format, then cooked down to more efficient representations for specific needs.

This is still experimental but I'm convinced that Haskell has a lot to offer for games development, and its tools for abstraction can make a lot of the complicated pieces of game development much easier to work with and reason about.

A quick way to run Postgres in Docker with local directory data storage

There are a lot of times where I want to quickly spin up a postgres server for some testing.

Unfortunately this usually involves installing it in the package manager and working with the global postgres config. Running postgres in Docker is better but volumes are tricky to deal with, and using data containers means you have to keep track of an extra container, as well as remembering to clean up volumes that are no longer used.

So I wrote this quick script that does all the steps needed to run a postgres docker container using a local directory for data storage, instead of volumes. So instead of having to leave your TestWebAppPostgresDb container and TestDataProcessingPostgresDb container hanging around in docker you can just leave them in local directories.

Quick usage

#Build the docker image
sudo docker build -t stevechy/postgres:v9.3 [DirectoryWithDockerFile]

#Make a database directory
mkdir -p projects/testDatabases/mytestdb

#Copy runDockerPostgres.py to the directory, or symlink it
cp runDockerPostgres.py projects/testDatabases/mytestdb

#Grab the default /etc/postgresql config from the docker image
cd projects/testDatabases/mytestdb
./runDockerPostgres.py --action extractEtcPostgresql

#Configure etc_postgresql/9.3/main/postgresql.conf if needed, for example, setting the port

#Initialize the postgres data cluster
./runDockerPostgres.py --action initDb

#You'll need a running postgres to create databases and users, start postgres in another terminal
./runDockerPostgres.py --action run

#Create a user
./runDockerPostgres.py --action createuser
#Enter name of role to add: testuser
#Enter password for new role: 
#Enter it again: 
#Shall the new role be a superuser? (y/n) n
#Shall the new role be allowed to create databases? (y/n) n
#Shall the new role be allowed to create more new roles? (y/n) n

#Create a database
./runDockerPostgres.py --action createdb --dbowner testuser --dbname testdb

You should now have a local running postgres that you can connect to.

Dockerfile:

from debian:wheezy

RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ wheezy-pgdg main" > /etc/apt/sources.list.d/pgdg.list
RUN apt-get update

RUN apt-get -y install wget ca-certificates

RUN wget --quiet https://www.postgresql.org/media/keys/ACCC4CF8.asc ; apt-key add ACCC4CF8.asc

RUN apt-get update

RUN apt-get -y install postgresql-9.3

EXPOSE 5432

USER postgres

CMD ["/bin/bash"]

runDockerPostgres.py