Tips and Tricks to use Unity with the Intel RealSense SDK 2014

This blog is meant to provide information, suggestions, and links for developers using Unity with the Intel RealSense SDK 2014. Please feel free to add info or corrections in the comments and I will add those into this article in future versions. Check back often for additions.

First - existing Documentation that may be helpful.

Intel provided: The RSSDK Unity Toolkit guide is installed at C:\Program Files (x86)\Intel\RSSDK\doc\PDF\sdktoolkit.pdf
- Unity_Getting_Started
- Face Tracking tutorial includes face location, pose, landmark, and expression detection, face recognition, alerts and code samples
- Hand/Joint/Blob Tracking tutorial covers gesture, hand and joint tracking, blob and contour extraction, data smoothing & code samples
- Raw Stream Capture tutorial :Catching and rendering color and depth streams with code samples
- Unofficial - Intel developer Amit M has Unity Toolkit Demo and RS/Unity RFBDepth Alignment videos
Marty G of Sambiglyon has shared these guides:
- Setting up the SDK files in Unity and programming objects in the 'Inspector' panel to respond to camera inputs.
- Converting PXC scripts for use in Unity script files
- Prototyping RealSense project controls with physical controllers using Unity's built-in "Input Manager".

It should be noted that one can use the Unity DLL without using the Unity Toolkit. If you post questions or issues, please distinguish between the two.

Some Minimum Requirements:

With Gold R1 ONLY Windows 8.1 is supported. The only camera supported is Creative RealSense F200 USER facing camera, which must be in a true (blue) USB 3.0 direct port (not hubbed).
Need to use Unity Pro because the free version of Unity doesn't support C++ plugins. There may exist workarounds to this, but they are not endorsed by Intel. Even as part of the contest, Intel is NOT providing any Pro licenses. But note that RSSDK 2014 can use C# and comes packaged with a lot of pre-built "extensions" for interfacing the camera with Unity. (thanks Marty!)

Now some tips/tricks:

Full Screen: Make the App go Full Screen in windowed/unwindowed modes, whatever the display resolution.
- Note Intel is suggesting a minimum resolution of 1368x768
- Before compiling, in Player’s Settings, make sure the "Use Direct3D 11" box is checked.
  - If only D3D9 is available in Player Settings, Uncheck "Default is Native Resolution" or "Default is Full Screen" and set default screen width and height to 1368x768
- Set it to use ONLY specific aspect ratios: 16:9 and 16:10
- Result: Based on the "Windowed" Checkbox setting on the "Resolution Selection Screen"
  - If checked, App will start windowed but can be resized/maximized
  - If unchecked, App will always start at full screen.
- Note you can check resolution // Cap screen resolution, an example
  if ( Screen.currentResolution.width > 1366 || Screen.currentResolution.height > 768 ) Screen.SetResolution(1366, 768, true);
Performance:
- The AcquireFrame desired setting in the Unity Update() function should be (false,0) to avoid impacting performance. (In Unity, it is a blocking call which locks the frame rate to RS and is used to wait until any frame data is available)
  if (sm.AcquireFrame (false) != pxceStatus.PXCM_STATUS_NO_ERROR)
- While 2 hands tracking may be in the same thread, it is suggested you create separate thread for face and voice or any multi-modal use.
Smoothing in Unity: as in the SDK, 4 types of smoothing are available. Here's SUGGESTIONS for their use in Unity
- Stabilize smoothing: good for UX as it keep existing location unless a large movement is detected
- Weighted: Allows averaging the data according to any set of weights (Gaussian, Uniform, etc.) . Good for hands
- Quadratic: Time based smoothing which acts as stabilizer and smoother.
- Spring: Time based but has a linear effect on smoothed data
Voice/Speech in Unity. Unity can do Commands, and Speech to text (can use the C# wrapper sample from framework folder) Speech synthesis may be doable the same way (use sample from C#) .
3D Segmentation - Beards. Currently long beards may cause very slow detection as the beard absorbs a lot of IR and may separate the head from body, confusing detection. The issue is being worked.
Face Recognition Registration.
- In Gold R1, 2D face recognition is only using ID#100.
- Gold changed from Beta and now waits 10 consecutive frames before confirming recognition (not settable since used to lessen false positives.
If you have issues with UnityEngine.Texture2D PXCMImage.ImageData.ToTexture2D(Int32 plane, Int32 width, Int32 height)),
use ToTexture2D that specifies a Texture2D as a parameter.