Learn how to display 3D graphics on SmartEyeglass

Did you know that SmartEyeglass can display advanced 3D graphics, overlaid on the user’s field of vision? In this tutorial Sony Software Engineer Ahmet Yildirim describes how this can be done using Open Graphics Library (OpenGL). Ahmet has also created a 3D model viewer sample app that we are making available as open source.

Ahmet Yildirim, Software Engineer at Sony.

Ahmet Yildirim, Software Engineer at Sony.

In this tutorial you will learn about the 3D model viewer project for SmartEyeglass, created by Sony Software Engineer Ahmet Yildirim. This tutorial and project will show you how to implement 3D graphics in your own SmartEyeglass app.

Images displayed in SmartEyeglass are based on Android’s layout system of bitmap images. However, it’s also possible to set up SmartEyeglass to display advanced 3D graphics by implementing OpenGL in your application. If you’ve implemented OpenGL in your app, you can then load and display 3D model files rendered as bitmaps.

OpenGL is a widely used open source API for rendering vector graphics, and using it can serve many different use cases where you would like to display complex graphic structures such as controllable and rotatable 3D models, dynamic graphics, and animations. Thanks to the sensors in SmartEyeglass, it’s also possible to control and rotate the 3D model.

In the following sections you will learn how to:

  • Import the 3D model viewer project to Eclipse.
  • Use OpenGL to render bitmap images.
  • Rotate a 3D model using the rotation vector sensor.
  • Customize the preference activity screen.

Import the 3D model viewer project to Eclipse

The following instructions require that you are familiar with Android app development, and that you have the SmartEyeglass SDK installed on your computer. Learn how to install the SmartEyeglass SDK in the SmartEyeglass get started guide. The SmartEyeglass SDK includes an emulator which you will need to install on your phone. How this is done is described in the SmartEyeglass test and verify page.

The 3D model viewer sample app is available as a downloadable Eclipse project. To import the project and add a few extra required packages to your development workspace, follow the steps below.

  1. Download the 3D model viewer zipped archive to your computer. Extract it and save the directories (3DModels and 3DModelViewer-Eclipse) to your computer.
  2. In Eclipse, import the directory 3DModelViewer-Eclipse to your workspace.
  3. Continue to import SmartExtensionAPI, SmartEyeglassAPI, and SmartExtensionUtils from your SmartEyeglass SDK installation to your Eclipse workspace.
  4. Download aFileDialog, a simple and easy to use open source file manager shared on Google Code. Save it to your computer, and import it to your workspace in Eclipse.
  5. In Eclipse, right-click your 3DModelViewer project and select Properties.
  6. In the properties window, click Android in the left pane. In the top right field, make sure that Android 4.4.2 is selected as Project Build Target. In the bottom right field, add the imported libraries (SmartExtensionAPI, SmartEyeglassAPI, SmartExtensionUtils and aFileDialog). Then click OK.

    The Properties window for the 3D model viewer project in Eclipse.

    The Properties window for the 3D model viewer project in Eclipse.

Before moving on to test the 3D model viewer, you should add some 3D model files to the sample app. Learn how to do this in the next section.

Adding 3D models to the 3D model viewer
The 3D model viewer sample app supports 3D models in OBJ file format. You will find two examples in your 3DModels directory (from step 1 above).

Follow the steps below to add the 3D models by using the SmartEyeglass emulator:

  1. Connect your phone to your computer using a micro USB cable.
  2. Start the SmartEyeglass emulator on your phone.
  3. In the SmartEyeglass emulator, select Settings from the top right corner menu (the three vertical dots).
    SmartEyeglass emulator displaying a 3D model.

    SmartEyeglass emulator displaying a 3D model.

    Note: If you are running the SmartEyeglass device, you can reach Settings from either the Notification Bar or the SmartConnect app on your phone.

  4. Go to Applications and then select 3D Model Viewer > 3D Model Files > Add Model File.
  5. Select the OBJ model file you want to add. Note: Make sure that the corresponding texture image file in JPG format is located in the same directory.
  6. If you would like to add your own 3D models, check the model exporting tips in Settings > Applications > 3D Model Viewer > 3D Model Files > Help.

    The preference activity screen of the 3D model viewer. From here you can select Add Model File to add new 3D model files, or Help to learn more about the application. In this example, two 3D files are added and listed on the page.

    The preference activity screen of the 3D model viewer. From here you can select Add Model File to add new 3D model files, or Help to learn more about the application. In this example, two 3D files are added and listed on the page.

Running the 3D model viewer sample app
To run the 3D model viewer sample app for the first time, follow the steps below.

  1. Connect your phone to your computer using a micro USB cable.
  2. In Eclipse, compile and run the 3D model viewer project by selecting Run from the menu Run. Doing so, the sample app will automatically be installed on your phone.
  3. Start the SmartEyeglass emulator on your phone, and navigate through your apps by swiping left or right until you find 3D model viewer.
  4. Tap to start the application.

When the sample app is running you will see the first 3D model that you added, rendered and shown on the SmartEyeglass display. You can navigate through your different models by swiping left or right. If you are using the emulator, you can rotate your phone to look around the 3D model. This corresponds to turning your head to look around the 3D model if you are wearing the SmartEyeglass device. To reset the rotation of the model, simply tap the touchpad sensor.

In the following sections you will learn how to use OpenGL to render bitmap images, and to rotate those and to display them on SmartEyeglass.

How to use OpenGL to render bitmap images

In order to display 3D models on SmartEyeglass, the rendered frames from OpenGL needs to be converted into bitmap images by PixelBuffer. This can be done using a regular Android GLSurfaceView.Renderer instance or, as shown below, the ModelRenderer. The bitmap images can then be displayed on SmartEyeglass.

To implement this method in your own project you should declare and set the OpenGL version by adding the following line in your AndroidManifest.xml file:

<uses-feature android:glEsVersion="0x00010000" android:required="true"></uses-feature>

Please note that the following classes of the 3D model viewer are used in the code examples below.

PixelBuffer This class handles OpenGL context and rendering frames into bitmaps.
ModelRenderer This is the renderer class for the PixelBuffer. It acts as a normal GLSurfaceView.Renderer.
DrawModel This class parses and renders 3D model files in OBJ file format.
GLRenderControlSEG This is the normal ControlExtension class for SmartEyeglass. It fetches bitmaps from PixelBuffer and displays them on SmartEyeglass, after rotating them according to sensor values.
GLRenderActivity This is the settings activity of the application. It’s where you can manage your different models in the application.
GLRenderExtensionService The ExtensionService class is responsible for creating control extensions for different platforms. It also keeps references to those controllers to enable interaction between them.

Overview of the classes used in the 3D model viewer.

How to retrieve a single frame bitmap
The following steps explain how to retrieve and display a single frame bitmap on SmartEyeglass. The code should be implemented in the ControlExtension class of your project.

  1. Get the file path for the 3D model file (*.obj) and texture file (*.jpg):
    String ModelFile = files.get(modelCursor);
    String TextureFile = ModelFile.replace(".obj", ".jpg");
  2. Create a ModelRenderer instance for the selected 3D model:
    mModelRenderer = new ModelRenderer(context,ModelFile,TextureFile);
  3. Initialize PixelBuffer for OpenGL and set the renderer instance:
    pixelBuffer=new PixelBuffer(surfaceWidth,surfaceHeight);
    pixelBuffer.setRenderer(mModelRenderer);
  4. Finally, let PixelBuffer render the frame as a bitmap and send it to SmartEyeglass:
    utils.showBitmap(pixelBuffer.getBitmap(),surfaceX,surfaceY);

This is how you can convert an OpenGL rendered frame into a single, static bitmap to display on SmartEyeglass. To display dynamic graphics and animations you will need a way to do this repeatedly, with continuous frame updates.

How to set up continuous frame updates
To render animations or dynamic graphics, like the 3D model in the 3D model viewer, you can implement a scheduled timer to call for continuous frame updates. How this is done is described in the steps below.

  1. In the ControlExtension class, initialize the Timer instance and set it to call UpdateCaller every 200 ms (5 FPS) with an initial 200 ms delay:
    frameUpdateTimer = new Timer();
    frameUpdateTimer.schedule(new UpdateCaller(new Handler(), this), 200, 200);
  2. Call for a frame update from the TimerTask. The TimerTask requires a reference back to the ControlExtension class because it cannot interact directly with the OpenGL instance. In the 3D model viewer, the call for a frame update is done as in the following code example:
    private class UpdateCaller extends TimerTask {
      Handler handler;
      GLRenderControlSEG ref;
      public UpdateCaller(Handler handler, GLRenderControlSEG ref) {
        super();
        this.handler = handler;
        this.ref = ref;
      }
            
      @Override
      public void run() {
        handler.post(new Runnable() {                
        @Override
        void run() {
          ref.update();
        }
        });
      }    
    }
  3. Finally, let PixelBuffer render the frame as a bitmap and send it to SmartEyeglass:
    // Requests for a new frame render and sends the rendered frame to SmartEyeglass
    public void update()
    {
      utils.showBitmap(pixelBuffer.getBitmap(),surfaceX,surfaceY);
    }

Note: Since each frame will be sent via Bluetooth, or Wi-Fi when using high power mode, you should make sure to set the frames per second (FPS) as low as possible to avoid draining the battery. The size of the individual bitmaps will also affect the performance of the Bluetooth transmission. If you would like to achieve high FPS, you should try to use a smaller surface area. In the 3D Model Viewer, 138 x 138 pixels was chosen to keep the size inside the boundaries of the SmartEyeglass 419 x 138 pixels display.

Coming this far you have learned how to use the OpenGL PixelBuffer to render bitmaps that can be displayed on SmartEyeglass. You have also set up a timer that calls for an update every 200 ms. However, if the 3D model isn’t turned or moved, the same frame will be updated over and over again. In the next section you will therefore learn how to rotate your model using SmartEyeglass sensor data.

How to rotate the 3D model using the rotation vector sensor

Rotating your 3D model is basically done in three steps:

  • Collecting rotation vector sensor data.
  • Converting the data to orientation angles.
  • Applying the orientation angles to the 3D model.

Collect rotation vector sensor data
To collect the rotation vector data you first need to register the sensor to a listener instance, and then pass the data on for further calculation. This is done as in the following steps.

  1. In the ControlExtension class, register the rotation vector sensor to a listener instance:
    mRotationVectorSensor.registerFixedRateListener(listener,Sensor.SensorRates.SENSOR_DELAY_FASTEST);
  2. The sensor listener instance should then pass on the sensor data to the sensorDataOperation for calculation:
    // Listens and passes incoming sensor data to the sensorDataOperation method
    private final AccessorySensorEventListener listener = new AccessorySensorEventListener() {
      @Override
      public void onSensorEvent(final AccessorySensorEvent sensorEvent) {
        sensorDataOperation(sensorEvent);           
      }
    };

Convert the data to orientation angles
The 3D model viewer uses glRotatef to rotate the 3D model. Since glRotatef requires orientation angles, you need to convert the rotation vector values to orientation angles and store them in the orientationVector. This is done as:

// Handles RotationVector sensor data
public void sensorDataOperation(AccessorySensorEvent event)
{
  // Convert RotationVector from Quaternion to Euclidian geometry
  float[] quat = new float[4];
  SensorManager.getQuaternionFromVector(quat, event.getSensorValues());
  float[] switchedQuat = new float[] {quat[1], quat[2], quat[3], quat[0]};
  float[] rotationMatrix = new float[16];
  SensorManager.getRotationMatrixFromVector(rotationMatrix, switchedQuat);
  SensorManager.getOrientation(rotationMatrix, orientationVector);
    
  // Check if heading update is requested
  if(updateHeading)
  {
    headingVector=orientationVector.clone();
    updateHeading=false;
  }
    
  // Rotate object by subtracting current sensor values from the initial orientation
  orientationVector[0]=orientationVector[0]-headingVector[0];
  orientationVector[1]=orientationVector[1]-headingVector[1];
  orientationVector[2]=orientationVector[2]-headingVector[2];
    
  // Convert Radian values to Degrees
  orientationVector[0]=(float) Math.toDegrees(GLRenderControlSEG.orientationVector[0])*rotationFactor;
  orientationVector[1]=(float) Math.toDegrees(GLRenderControlSEG.orientationVector[1])*rotationFactor;
  orientationVector[2]=(float) Math.toDegrees(GLRenderControlSEG.orientationVector[2])*rotationFactor;
}

Apply the orientation angles to the 3D model
Depending on the sensor data, the 3D model can be rotated on the X, Y and Z axes. The X and Z axes are initially set to -1, because the 3D model should rotate in the opposite direction of the sensor values to compensate for the orientation of the user’s head.

To do this you should call glRotatef in the onDrawFrame method in the renderer (ModelRenderer) class, as in the following code:

gl.glRotatef(GLRenderControlSEG.orientationVector[1]+angleOffsetX,-1,0,0);
gl.glRotatef(GLRenderControlSEG.orientationVector[0]+angleOffsetY,0,1,0);
gl.glRotatef(GLRenderControlSEG.orientationVector[2]+angleOffsetZ,0,0,-1);

To sum it up, new sensor data has been converted into new angles which have been used to rotate the 3D model, and finally resulting in new frames being rendered and displayed on SmartEyeglass.

How to customize the preference activity screen

The preference activity screen is accessible from the Settings menu in SmartEyeglass. In the 3D model viewer the preference activity screen is a file manager that handles the 3D model files. The preference activity can be customized just like any other activity.

To display your own layout design, you need to make a few changes to the preference activity. This is done as:

  1. Extend the GLRenderActivity class from PreferenceActivity to Activity:
    // This sets the settings activity to a normal activity instead of PreferenceActivity
    public final class GLRenderActivity extends Activity {
  2. Set up your custom layout in the onCreate method of the Activity class:
    @Override
    public void onCreate(Bundle savedInstanceState) {
      super.onCreate(savedInstanceState);
      setContentView(R.layout.activity_main);
    }

Check out the GLRenderActivity of the 3D model viewer application to see a more detailed implementation of this approach. You can also find more information about the preference activity on the Android Developer site.

Done! After following the step-by-step instructions in this tutorial, and using the 3D model viewer as an example, you should now be familiar with how to implement and use OpenGL in your application to display advanced 3D graphics on SmartEyeglass. You should also know how to customize the preference activity screen of your application.

***

Hopefully this tutorial has made you think about how you can implement the described technique in new and innovative use cases. If you have any comments or questions on this subject, feel free to drop us a line below!

More information

Comments 0

Sort by:

Showing 0 of 0 comments. Show all comments