In this post I would like to share with you my thoughts and ideas behind mouse selection. This is important technique for any interactive apps.

There are several ways of doing the picking in OpenGL:

  • Using selection/feedback mode in OpenGL. Since OGL3.0 this feature is deprecated. I suggest not using it.
  • Using color coding: render objects in different color and then read color from framebuffer render mouse position.
  • Using mouse Ray. We cast a ray from the mouse position and we test what objects in the scene are hit. In this post I will write about that method.

So let’s get started!

Before we start, please look at our target for the article:

Read more to find how to create such application!

Idea  

We would like to create a ray, starting in mouse position at the near plane. Then we will test what objects collide with the ray.

Creation of the mouse ray  

At first we can easily obtain position of the mouse in 2D in window coordinates. Let us name it MX and MY. We can now use gluUnProject to convert that position from window coordinates to 3D scene position. gluUnProject needs:

  • window X pos
  • window Y pos - we need to “invert” our MY, because in OpenGL Y axis has direction that is reverse to window Y axis (in Windows for instance). So we simply use WindowHeight - MY
  • window Z pos - we would like to have a ray that travels through the whole scene. We can use z values from near and far plane (since everything outside those planes are clipped). It is quite simple: for the near plane z value is 0.0, and for the far plane z value is 1.0. See the code below for more explanation.
  • model view matrix - just get it from OpenGL (using glGet*)
  • projection matrix - just get it from OpenGL
  • viewport - just get it from OpenGL :)
  • output: objX, objY, objZ - calculated mouse position in the scene.

TODO: gluUnProject is deprecated. But it can be easily replaced by the glm lib for instance.
NOTE: instead of glGet* we can provide custom matrices, from our own math lib, like GLM.

Here are a few lines of code:

double matModelView[16], matProjection[16]; 
int viewport[4]; 
glGetDoublev( GL_MODELVIEW_MATRIX, matModelView ); 
glGetDoublev( GL_PROJECTION_MATRIX, matProjection ); 
glGetIntegerv( GL_VIEWPORT, viewport ); 
double winX = (double)mouseX; 
double winY = viewport[3] - (double)mouseY; 
gluUnProject(winX, winY, 0.0, matModelView, matProjection, 
             viewport, m_start.x, &m_start.y, &m_start.z); 
gluUnProject(winX, winY, 1.0, matModelView, matProjection, 
             viewport, m_end.x, &m_end.y, &m_end.z);

As you can see we use gluUnProject two times: one for the starting point (on the near plane), and the second time for the end point of the ray (on the far plane).

Collision between the ray and a sphere  

What we have right now is the mouse ray and a bunch of objects in the scene that we want to check if there is a collision between them and the ray.

As you can see in the picture the mouse ray hits yellow ball that is in the scene. Of course testing collision between the ray and a sphere is one of the simplest cases. It is a good starting point for more advanced collision tests: with AABB (Axis Aligned Bounding Boxes), OBB (Oriented Bounding Boxes) or even arbitrary meshes. On the other hand usually we have “hierarchy” of collision tests: from simple to more advanced… so testing with spheres is usually a must-have tool.

The test info:

Just find the distance between the ray (3d line) and the center of the sphere. If that distance is less then the radius of the sphere then we have a hit!_

Here is a bit of code for calculating the distance between line and a point:

// pseudo code found at: 
// http://www.gamedev.net/topic/221071-simple-raysphere-collision/ 
Vec3d ClosestPoint(const Vec3d A, const Vec3d B, 
                   const Vec3d P, double *t) 
{ 
    Vec3d AB = B - A; 
    double ab_square = DotProduct(AB, AB); 
    Vec3d AP = P - A; 
    double ap_dot_ab = DotProduct(AP, AB); 
    // t is a projection param when we project vector AP onto AB 
    *t = ap_dot_ab / ab_square; 
    // calculate the closest point 
    Vec3d Q = A + AB * (*t); 
    return Q; 
}

Basically, we project the AP vector (that starts at the given point) onto the AB vector (that starts at the end of the moue ray). T is a “projection” param, it should be between 0 and 1… If the point is “outside” the range (outside space between near and far plane), T will be outside this [0, 1] range.

Collision test:

bool RayTest(const Vec3d, const Vec3d start, const Vec3d end, 
                  Vec3d *pt, double *t, double epsilon) 
{ 
    *pt = ClosestPoint(start, end, center, t); 
    double len = Distance(*pt, m_pos); 
    return len < (m_radius+epsilon); 
} 
// note that "t" param can be used further 
// the same is with "pt"

Not so hard. Of course if you choose more advanced shapes this will complicate a bit…

The test placement  

Now we need to put that test in proper place. Ray calculation needs camera matrices. We can do our test in the Render function just after we set up our camera matrix.

void Render() 
{ 
    // clear screen ... 
    // setup camera ... 

    // HIT test: 
    CalculateRay(); 
    CheckHitTestBetweenPointsInTheScene();

    // render the scene... 
}

But wait…! Did I miss something?

We usually have several… or several thousands of objects, how to test the collision? Let us look inside the CheckHitTestBetweenPointsInTheScene.

bool RayTestPoints(const Vec3d &start, const Vec3d &end, 
     unsigned int *id, double *t, double epsilon)
{
    unsigned int pointID = m_count+1;
    bool foundCollision = false;
    double minDistToStart = 10000000.0;
    double dst;
    Vec3d pt;
    for (unsigned int i = 0; i < m_count; ++i)
    {
        if (m_points[i].RayTest(start, end, &pt, t, epsilon))
        {
            dst = Distance(start, pt);
            if (dst < minDistToStart)
            {
                minDistToStart = dst;
                pointID = i;
                foundCollision = true;
            }
        }
    }

    *id = pointID;

    return foundCollision;
}

Basically it is a simple brute force solution: run through all the points (spheres), find the closest one to the mouse ray and then treat this object as a hit. Please notice that although the ray can hit several objects, we choose only one.

Dragging point by the mouse  

OK We have a hit! What to do next with that? We need to handle the mouse input: detect if we are dragging a point and finally move the point. Here is a bit of code for dragging:

void ProcessMouse(int button, int state, int x, int y) 
{ 
    g_camera.ProcessMouse(button, state, x, y); 
    // can we move any ball? 
    if (g_camera.m_isLeftPressed && g_we_got_hit) 
    { 
        g_draggedPointID = g_lastHitPointID; 
        g_areWeDraggingPoint = true; 
    } 
    else  
        g_areWeDraggingPoint = false; 
}

We process mouse buttons and when LBM is pressed and we detected a hit… then we can drag that point. Then in the Update proc we can change position of the point. Remember that we can move point only on the plane parallel to the camera front plane. We save “T” param from the hit test (from the line equation), and when we move mouse we use that “T” param to calculate new position in the 3D space.

Vec3d v2 = m_end - m_start; 
Vec3d point_in_3d = m_start + v2*m_lastT; 
// m_lastT is "T" param found in hit test

now in the Update proc we can move point by simply:

our_model_pos = point_in_3d;

Or better way: We could use some spring system equation.. but this is your homework.

Source Code & Sample App  

References: