Tips and Tricks for developing Apps with the Intel RealSense SDK 2014 R1

Here's a collection of tips and tricks when using the Intel RealSense Gold R1 2014 SDK.

Performance Tuning:

1. ​Turn off unused modalities, only call the specific functions required. Especially check multiple modalities within hands or face such as hand_facade, blob/contour, or second hand usage. (Example: May not need hand tracking during gesture use).
2. Use separate threads for different modalities (except 2 hands gestures). This will help in only running required modality for each section. But remember switch to thread can be CPU intensive.
3. Specify the gestures actually used and only look for the specified gestures (gesture classification).
    
4. Only track the required joints/landmarks (might only need tip of nose or one finger joint.

   Finger Joint Tracking:  (thanks Marty!) To add stability (less twitch), limit the number of joints you track based on these tips:
   Bent Fingers are interdependent:  Use only two tracked joints and still make the hand's four fingers.bend realistically. Only the index finger can be bent individually, or remain unbent while other fingers bend - so it needs its own unique joint tracking. But for other fingers: .  Bend the middle finger then the ring and pinky are pulled down; 
    Bend the ring finger, the middle and pinky finger are pulled down.  
    Bend the pinky finger, the middle and ring fingers are pulled down. 
   Rotation: The degrees that a particular section of human finger can rotate is determined by the length of the section (not the joint).  The finger-tip of the middle finger, can bend less than the finger's middle section, and the middle section bends less than the base section.  You only need a TrackingScript cover Index: Joint_Index_IT1 in all joints; Middle, Ring and Pinky fingers: Joint_Middle_ITI in all joints.
   Children's Hands: Small hands are difficult to acquire. In Gold R1, please allow 5 seconds to lock in and acquire. 

5. Pause the camera when it’s not needed
6. Cap the App’s FPS rate (30/60) to help synchronize with slower RSSDK FPS, at least when the camera is active.
7. Disassociate the game from RSSDK frames (don’t wait for frames).
8. Run you app and look for any sections that are GPU or CPU bound.
9. Remember that ID’ing of a hand is slow and intensive– so work to avoid it. Warn users if they’re about to leave FOV, Warn users to slow motions down; etc.
10. Cache when possible to avoid component lookup overhead. Using persistent storage avoids creating data structure and objects every frame.
11. Reduce screen resolution (can give up to 30% fps increase) - Intel requesting 1380x
12. Don't mix 32 and 64 bit libraries
13. Ways to lower high GPU usage:  Render close objects first; Reduce Textures; and Limit floating point calls

Which Modality to Use Tips:

• Contour Mode requires skeleton, can’t use hand openness.  
• Head Movement (pitch, yaw, roll)  - use Pose Detection from Face tracking. 
• Navigation/Cursor replacements - TouchlessController module is specifically designed for this.

Usability Tips

1. Smoothing: Easiest method is often frame averaging (suggest no more than 25 frames running average).
   Note: Unity toolkit includes good smoothing.
   Use the best of the 4 smoothing types (all of which can be 1D, 2D, or 3D) for your actions:
       Stabilize – Best for UI as keeps current location until large movement
       Quadratic – Good for UI stabilization as it’s time based.
       Weighted – Allows different weight per section (Gaussian, Uniform, etc.)
       Spring – Best for gaming – linear affect based on time.
   Note: Hands and contour extraction smoothing use 0-1(max) scale.
             On Spring/Quadratic – the default setting is .5
   Ensure all instances are disposed of,  before disposing of the PXCMSenseManager.
   With hands, config->EnableStabilizer(true) is on by default but can be set to false.
2. Multiple gestures - Don't use similar gestures together (for example: thumbs up/down and fist). 
3.  See the Design Guidelines https://software.intel.com/sites/default/files/managed/0b/0a/RealSenseSDKdesignGuidelinesGold.pdf

Camera Settings:

1. To set camera values found in manuals_member_functions23.html, includingSetColorAutoWhiteBalance and SetColorAutoExposure, you must add the parameter auto = false so that the manual setting is picked up. Also both the item_unavailable and device_failed error messages are expected on errors. ,

2. Default camera settings were aimed at general color and depth. As you can see in the module samples, you can add camera settings to your code that will help with differences between modules such as fast movement or segmentation such as:   device.SetDepthConfidenceThreshold(1); device.SetIVCAMFilterOption(6);
   For optimal performance Intel suggests:

Known Issues:

• If some of the samples aren't working, make sure your user folder name contains only characters in the English alphabet
• Java: only HandsViewer works, not raw stream.
• Color to Depth Mapping: Memory leak in PXCProjection::CreateColorImageMappedToDepth. So use DepthtoColor Mapping.

For more tips and tricks, search the RealSense SDK Forum for specific topics.