3D Object Detection From Mouse Click

Published 2018-08-06 13:55
Tagged on 3D

We often use mouse event in computer graphics. I’m pretty sure that we can easily detect a 2D object using mouse event in OpenGL since a view-port or screen is of 2D. But problem arises whenever we have 3D space model having perspective projection and we are asked to detect an object using a mouse click. Ok! Let’s consider an example. Suppose, we have an sphere shaped object locating at (0,0,0) and the radius is 100 unit in our model. After doing some work on perspective projection we are done with setting up camera position, object position, projection angle etc and now see the effect. What are we watching? Let’s assume the object is now located at right-top (640,480) corner of our view screen and our view screen is of 640×480. What actually we are seeing?  *Figure: Sphere at (0,0,0) originally*  *Figure Sphere at (640,480) from window screen* Now click on the sphere and use your mouse event. The co-ordinate you are getting is simply something close around (640,480) and how would I say this clicked object is actually situated at (0,0,0)? So, we would like to confirm that this clicked upon object is surely the object located at (0,0,0) of our main 3D space. That means, we are going to find out the main object from this 2D point. See the following picture and try to feel what we are going to do now.  *Figure: Perspective projection and mouse click* Can you see a line is drawn from clicked point to the view volume? We can only find this line using gluUnProject() function given in OpenGL. For better understanding see the following image too.  *Figure: Line we can detect using gluUnproject()* Now see, how the function works. Usually for projection what we do **3D Pont World -> Model View -> Projection -> Viewport -> 2D Point on Screen** So what we have to do is just reverse the process- **2D Point on Screen -> Viewport-1 -> Projection-1 -> Model View-1 -> 3D Point World** Inverse is nothing but simply the inverse of the matrices. gluUnProject does this task for us. Let’s see the function parameters. ``` GLint gluUnProject(GLdouble winX, GLdouble winY, GLdouble winZ, const GLdouble * model, const GLdouble * proj, const GLint * view, GLdouble* objX, GLdouble* objY, GLdouble* objZ); ``` Parameters are used for the purposes respectively: - window X pos - window Y pos – we need to “invert” our **MY**, because in OpenGL Y axis has direction that is reverse to window Y axis (in Windows for instance). So we simply use *WindowHeight – MY (See example code)* - window Z pos – [OMG! what parameter? Z? How come? How a window screen has Z point? Not actually. It’s for assuming the points which are on the ray created when you have put a click on the screen (shown in the above pictures)] we would like to have a ray that travels through the whole scene. We can use z values from near and far plane (since everything outside those planes are clipped). It is quite simple: for **the near plane z value is 0.0**, and for **the far plane z value is 1.0**. *[See the code below for more explanation]* - model view matrix – just get it from OpenGL *(we get it using glGetDouble, see example )* - projection matrix – just get it from OpenGL - viewport – just get it from OpenGL; infact we defined it earlier - output: **objX, objY, objZ** – calculated mouse position in the scene. Not all matrices are invertible thus, gluUnProject() might not succeed everytime, in which case it will return GL_FALSE, with successful inversion returning GL_TRUE. Let this aside now and come back to our main goal. It turns out that, the way OpenGL calculates things, winZ == 0.0 (the screen) corresponds to objZ == N (the near plane), and winZ == 1.0 corresponds to objZ == F (the far plane) . Since two points determine a line, we actually need to call gluUnProject() twice: once with winZ == 0.0, then again with winZ == 1.0 this will give us the world points that correspond to the mouse click on the near and far planes, respectively. Now see the snippet of corresponding function calls reagrding these. ``` double matModelView[16], matProjection[16]; int viewport[4]; int nx, ny, nz; //declared for near points int fx, fy, fz; //declared for far points glGetDoublev( GL_MODELVIEW_MATRIX, matModelView ); glGetDoublev( GL_PROJECTION_MATRIX, matProjection ); glGetIntegerv( GL_VIEWPORT, viewport ); double winX = (double)mouseX; double winY = viewport[3] – (double)mouseY;//if viewport’s Y is 640 then you can write also, winY = 640 – (double)mouseY; gluUnProject(winX, winY, 0.0, matModelView, matProjection, viewport, &nx, &ny, &nz); gluUnProject(winX, winY, 1.0, matModelView, matProjection, viewport, &fx, &fy, &fz); ``` So, this is the code for getting the ray from (nx, ny, nz) to (fx ,fy, fz). Now, this is obvious that your 3d object must lies on the ray. So, just some checking can detect your object, only if you store the objects center points, radius previously. This is so simple. When you draw the object just store the center point along with radius or height etc . In our case, sphere has a radius and center point. You are learning Graphics and have come through a long way. So I can assume that you have already learned about parametric equation of a line. If you don’t know just go through your secondary level geometry book and come back to see whether the following equation looks like so (:p). > **L(t) = nearPoint + t(farPoint - nearPoint)** > > How it works? Set t = 0 and get the nearPoint (it is 0 indeed) and set t = 1 and get the farPoint (it is 1 indeed). Now we can have a for loop operation from 0 to 1 having some floating points interval (more intervals, more accurate your result is). While looping, we will just check if the intersecting point belongs to the sphere inside. How can we do this? This is so simple, find the distance from center of the sphere to the point and see whether it is less or equal to the radius of the sphere. Something like that:  *Figure: Ray intersects the sphere* ``` bool clickRayPoints(){ float tx, ty, tz; for(float t=0;t<=1;t+=0.0001){ tx = nx + t*(fx-nx); ty = ny + t*(fy-ny); tz = nz + t*(fz-nz); //cout << tx << " " << ty << " " << tz << endl; if(distance( cx,cy,cz, tx,ty,tz) <= r) //write this distance() function yourself return true; } } ``` This is how you detected if the clicked point is on the sphere or not. There are more many type of objects i.e. cylinder, cone etc. You can write codes for those also in the same fashion adding some other conditions. If you want to detect a point is intersecting a cylinder you have to check whether abs(tx-cx), abs(tz-cz) values are less than radius and abs(ty-cy) value is less than height/2 [consider the cylinder is standing on x-z plane]. So this all about detecting a 3D object from a mouse click. [There are also some other way like Vector but I didn’d get anything easier. So I started thinking some ohter way and all on a sudden got the way I have implemented earlier to detect the object using parametric equation from the ray generated from gluUnProject.] So, have fun. Enjoy Graphics!

---