Oculus Rift VR headset Developer guide
Reklamë
Reklamë
Oculus Rift Developer Guide
Version latest
2 | Introduction | Oculus Rift
Copyrights and Trademarks
©
2017 Oculus VR, LLC. All Rights Reserved.
OCULUS VR, OCULUS, and RIFT are trademarks of Oculus VR, LLC. (C) Oculus VR, LLC. All rights reserved.
BLUETOOTH is a registered trademark of Bluetooth SIG, Inc. All other trademarks are the property of their respective owners. Certain materials included in this publication are reprinted with the permission of the copyright holder.
2 | |
Oculus Rift | Contents | 3
Contents
LibOVR Integration
..............................................................................................
5
Initialization and Sensor Enumeration
.................................................................
6
Rendering to the Oculus Rift
.............................................................................
13
Advanced Rendering Configuration
..................................................................
32
VR Focus Management
......................................................................................
39
Oculus Guardian System
...................................................................................
42
Rift Audio
45
Oculus Touch Controllers
..................................................................................
49
SDK Samples and Gamepad Usage
..................................................................
64
Optimizing Your Application
.............................................................................
65
4 | Contents | Oculus Rift
Pairing the Oculus Touch Controllers
................................................................
89
Asynchronous SpaceWarp
.................................................................................
90
Mixed Reality Capture Setup Guide
..................................................................
91
Oculus Rift | LibOVR Integration | 5
LibOVR Integration
The Oculus SDK is designed to be as easy to integrate as possible. This guide outlines a basic Oculus integration with a C/C++ game engine or application.
We’ll discuss initializing the LibOVR, HMD device enumeration, head tracking, frame timing, and rendering for the Rift.
Many of the code samples below are taken directly from the OculusRoomTiny demo source code (available in
Oculus/LibOVR/Samples/OculusRoomTiny). OculusRoomTiny and OculusWorldDemo are great places to view sample integration code when in doubt about a particular system or feature.
Overview of the SDK
There are three major phases when using the SDK: setup, the game loop, and shutdown.
To add Oculus support to a new application, do the following:
1. Initialize LibOVR through ovr_Initialize.
2. Call ovr_Create and check the return value to see if it succeeded. You can periodically poll for the presence of an HMD with ovr_GetHmdDesc(nullptr).
3. Integrate head-tracking into your application’s view and movement code. This involves: a. Obtaining predicted headset orientation for the frame through a combination of the
GetPredictedDisplayTime and ovr_GetTrackingState calls.
b. Applying Rift orientation and position to the camera view, while combining it with other application controls.
c. Modifying movement and game play to consider head orientation.
4. Initialize rendering for the HMD.
a. Select rendering parameters such as resolution and field of view based on HMD capabilities.
• See: ovr_GetFovTextureSize andovr_GetRenderDesc.
b. Configure rendering by creating D3D/OpenGL-specific swap texture sets to present data to the headset.
• See: ovr_CreateTextureSwapChainDX andovr_CreateTextureSwapChainGL.
5. Modify application frame rendering to integrate HMD support and proper frame timing: a. Make sure your engine supports rendering stereo views.
b. Add frame timing logic into the render loop to obtain correctly predicted eye render poses.
c. Render each eye’s view to intermediate render targets.
d. Submit the rendered frame to the headset by calling ovr_SubmitFrame.
6. Customize UI screens to work well inside of the headset.
7. Destroy the created resources during shutdown.
• See: ovr_DestroyTextureSwapChain, ovr_Destroy, and ovr_Shutdown.
A more complete summary of rendering details is provided in the
section.
6 | Initialization and Sensor Enumeration | Oculus Rift
Initialization and Sensor Enumeration
This example initializes LibOVR and requests information about the available HMD.
Review the following code:
// Include the OculusVR SDK
#include <OVR_CAPI.h>
void Application()
{
ovrResult result = ovr_Initialize(nullptr);
if (OVR_FAILURE(result))
return;
ovrSession session;
ovrGraphicsLuid luid;
result = ovr_Create(&session, &luid);
if (OVR_FAILURE(result))
{
ovr_Shutdown();
return;
}
ovrHmdDesc desc = ovr_GetHmdDesc(session);
ovrSizei resolution = desc.Resolution;
ovr_Destroy(session);
ovr_Shutdown();
}
As you can see, ovr_Initialize is called before any other API functions and ovr_Shutdown is called to shut down the library before you exit the program. In between these function calls, you are free to create HMD objects, access tracking state, and perform application rendering.
In this example, ovr_Create(&session &luid) creates the HMD. Use the LUID returned by ovr_Create() to select the IDXGIAdapter on which your ID3D11Device or ID3D12Device is created. Finally, ovr_Destroy must be called to clear the HMD before shutting down the library.
You can use ovr_GetHmdDesc() to get a description of the HMD.
If no Rift is plugged in, ovr_Create(&session, &luid) returns a failed ovrResult unless a virtual HMD is enabled through RiftConfigUtil. Although the virtual HMD will not provide any sensor input, it can be useful for debugging Rift-compatible rendering code and for general development without a physical device.
The description of an HMD ( ovrHmdDesc) handle can be retrieved by calling ovr_GetHmdDesc(session).
The following table describes the fields:
Field
Type
ProductName
Manufacturer
VendorId
ProductId
Type ovrHmdType char[] char[] short short
Description
Type of the HMD, such as ovr_DK2 or ovr_DK2 .
Name of the product as a string.
Name of the manufacturer.
Vendor ID reported by the headset
USB device.
Product ID reported by the headset
USB device.
Oculus Rift | Initialization and Sensor Enumeration | 7
Field
SerialNumber
FirmwareMajor
FirmwareMinor
AvailableHmdCaps
DefaultHmdCaps
AvailableTrackingCaps
DefaultTrackingCaps
DefaultEyeFov
MaxEyeFov
Resolution
DisplayRefreshRate
Type char[] short short unsigned int unsigned int unsigned int unsigned int ovrFovPort[] ovrFovPort[] ovrSizei float
Description
Serial number string reported by the headset USB device.
The major version of the sensor firmware.
The minor version of the sensor firmware.
Capability bits described by ovrHmdCaps which the HMD currently supports.
Default capability bits described by ovrHmdCaps for the current HMD.
Capability bits described by ovrTrackingCaps which the
HMD currently supports.
Default capability bits described by ovrTrackingCaps for the current
HMD.
Recommended optical field of view for each eye.
Maximum optical field of view that can be practically rendered for each eye.
Resolution of the full HMD screen
(both eyes) in pixels.
Nominal refresh rate of the HMD in cycles per second at the time of
HMD creation.
The description of a sensor ( ovrTrackerDesc) handle can be retrieved by calling ovr_GetTrackerDesc(sensor). The following table describes the fields:
Field
FrustumHFovInRadians
FrustumVFovInRadians
FrustumNearZInMeters
FrustumNearZInMeters
Type float float float float
Description
The horizontal FOV of the position sensor frustum.
The vertical FOV of the position sensor frustum.
The distance from the position sensor to the near frustum bounds.
The distance from the position sensor to the far frustum bounds.
8 | Initialization and Sensor Enumeration | Oculus Rift
Head Tracking and Sensors
The Oculus Rift hardware contains a number of micro-electrical-mechanical (MEMS) sensors including a gyroscope, accelerometer, and magnetometer.
There is also a sensor to track headset position. The information from each of these sensors is combined through the sensor fusion process to determine the motion of the user’s head in the real world and synchronize the user’s view in real-time.
Once the ovrSession is created, you can poll sensor fusion for head position and orientation by calling ovr_GetTrackingState. These calls are demonstrated by the following code:
// Query the HMD for ts current tracking state.
ovrTrackingState ts = ovr_GetTrackingState(session, ovr_GetTimeInSeconds(), ovrTrue); if (ts.StatusFlags & (ovrStatus_OrientationTracked | ovrStatus_PositionTracked))
{
ovrPosef pose = ts.HeadPose.ThePose;
...
}
This example initializes the sensors with orientation, yaw correction, and position tracking capabilities. If the sensor is not available during the time of the call, but is plugged in later, the sensor is automatically enabled by the SDK.
After the sensors are initialized, the sensor state is obtained by calling ovr_GetTrackingState. This state includes the predicted head pose and the current tracking state of the HMD as described by
StatusFlags.
This state can change at runtime based on the available devices and user behavior. For example, the ovrStatus_PositionTracked flag is only reported when HeadPose includes the absolute positional tracking data from the sensor.
The reported ovrPoseStatef includes full six degrees of freedom (6DoF) head tracking data including orientation, position, and their first and second derivatives. The pose value is reported for a specified absolute point in time using prediction, typically corresponding to the time in the future that this frame’s image will be displayed on screen. To facilitate prediction, ovr_GetTrackingState takes absolute time, in seconds, as a second argument. The current value of absolute time can be obtained by calling ovr_GetTimeInSeconds. If the time passed into ovr_GetTrackingState is the current time or earlier, the tracking state returned will be based on the latest sensor readings with no prediction. In a production application, however, you should use the real-time computed value returned by
GetPredictedDisplayTime. Prediction is covered in more detail in the section on Frame Timing.
Oculus Rift | Initialization and Sensor Enumeration | 9
As already discussed, the reported pose includes a 3D position vector and an orientation quaternion. The orientation is reported as a rotation in a right-handed coordinate system, as illustrated in the following figure.
Figure 1: Rift Coordinate System
The x-z plane is aligned with the ground regardless of camera orientation.
As seen from the diagram, the coordinate system uses the following axis definitions:
• Y is positive in the up direction.
• X is positive to the right.
• Z is positive heading backwards.
Rotation is maintained as a unit quaternion, but can also be reported in yaw-pitch-roll form. Positive rotation is counter-clockwise (CCW, direction of the rotation arrows in the diagram) when looking in the negative direction of each axis, and the component rotations are:
• Pitch is rotation around X, positive when pitching up.
• Yaw is rotation around Y, positive when turning left.
• Roll is rotation around Z, positive when tilting to the left in the XY plane.
10 | Initialization and Sensor Enumeration | Oculus Rift
The simplest way to extract yaw-pitch-roll from ovrPose is to use the C++ OVR Math helper classes that are included with the library. The following example uses direct conversion to assign ovrPosef to the equivalent
C
++ Posef class. You can then use the
Quatf::GetEulerAngles<> to extract the Euler angles in the desired axis rotation order.
All simple C math types provided by OVR such as ovrVector3f and ovrQuatf have corresponding C++ types that provide constructors and operators for convenience. These types can be used interchangeably.
If an application uses a left-handed coordinate system, it can use the ovrPosef_FlipHandedness function to switch any right-handed ovrPosef provided by ovr_GetTrackingState, ovr_GetEyePoses, or ovr_CalcEyePoses functions to be left-handed. Be aware that the RenderPose and
QuadPoseCenterrequested for the ovrLayers must still use the right-handed coordinate system.
Position Tracking
The frustum is defined by the horizontal and vertical FOV, and the distance to the front and back frustum planes.
Approximate values for these parameters can be accessed through the ovrTrackerDesc struct as follows: ovrSession session; ovrGraphicsLuid luid; if(OVR_SUCCESS(ovr_Create(&session, &luid)))
{
// Extract tracking frustum parameters.
float frustomHorizontalFOV = session->CameraFrustumHFovInRadians;
...
The following figure shows the tracking sensor and a representation of the resulting tracking frustum.
Figure 2: Tracking Sensor and Tracking Frustum
The relevant parameters and typical values are list below:
Field
FrustumHFovInRadians
FrustumVFovInRadians
FrustumNearZInMeters
FrustumFarZInMeters
Type float float float float
Typical Value
1.292 radians (74 degrees)
0.942 radians (54 degrees)
0.4m
2.5m
Oculus Rift | Initialization and Sensor Enumeration | 11
These parameters provide application developers with a visual representation of the tracking frustum. The previous figure also shows the default tracking origin and associated coordinate system.
Note: Although the sensor axis (and hence the tracking frustum) are shown tilted downwards slightly, the tracking coordinate system is always oriented horizontally such that the axes are parallel to the ground.
By default, the tracking origin is located one meter away from the sensor in the direction of the optical axis but with the same height as the sensor. The default origin orientation is level with the ground with the negative axis pointing towards the sensor. In other words, a headset yaw angle of zero corresponds to the user looking towards the sensor.
This can be modified using the API call ovr_RecenterTrackingOrigin, which resets the tracking origin to the headset’s current location and sets the yaw origin to the current headset yaw value. Additionally, it can be manually specified to any location using the API call ovr_SpecifyTrackingOrigin.
There are two types of tracking origins: floor-level and eye-level. Floor-level is recommended for room scale, when the user stands, and when the user is interacting with objects on the floor (although telekinesis/force grab/gaze grab is a better option for picking up objects). For most other experiences, especially when the user is seated, eye-level is preferred. To get the current origin, use ovr_GetTrackingOriginType. To set the origin, use ovr_SetTrackingOriginType.
Note: The tracking origin is set on a per application basis; switching focus between different VR apps also switches the tracking origin.
The head pose is returned by calling ovr_GetTrackingState. The returned ovrTrackingState struct contains several items relevant to position tracking:
•
HeadPose—includes both head position and orientation.
•
Pose—the pose of the sensor relative to the tracking origin.
•
CalibratedOrigin—the pose of the origin previously calibrated by the user and stored in the profile reported in the new recentered tracking origin space. This value can change when the application calls ovr_RecenterTrackingOrigin, though it refers to the same location in real-world space. Otherwise it will remain as an identity pose. Different tracking origin types will report different CalibrateOrigin poses, as the calibration origin refers to a fixed position in real-world space but the two tracking origin types refer to different y levels.
The
StatusFlags variable contains the following status bits relating to position tracking:
• ovrStatus_PositionTracked—flag that is set only when the headset is being actively tracked.
There are several conditions that may cause position tracking to be interrupted and for the flag to become zero:
• The headset moved wholly or partially outside the tracking frustum.
• The headset adopts an orientation that is not easily trackable with the current hardware (for example facing directly away from the sensor).
• The exterior of the headset is partially or fully occluded from the sensor’s point of view (for example by hair or hands).
• The velocity of the headset exceeds the expected range.
Following an interruption, assuming the conditions above are no longer present, tracking normally resumes quickly and the ovrStatus_PositionTracked flag is set.
If you want to get the pose and leveled pose of a sensor, call ovr_GetTrackerPose. The returned ovrTrackerPose struct contains the following:
•
Pose—the pose of the sensor relative to the tracking origin.
12 | Initialization and Sensor Enumeration | Oculus Rift
•
LeveledPose— the pose of the sensor relative to the tracking origin but with roll and pitch zeroed out. You can use this as a reference point to render real-world objects in the correct place.
User Input Integration
To provide the most comfortable, intuitive, and usable interface for the player, head tracking should be integrated with an existing control scheme for most applications.
For example, in a first person shooter (FPS) game, the player generally moves forward, backward, left, and right using the left joystick, and looks left, right, up, and down using the right joystick. When using the Rift, the player can now look left, right, up, and down, using their head. However, players should not be required to frequently turn their heads 180 degrees since this creates a bad user experience. Generally, they need a way to reorient themselves so that they are always comfortable (the same way in which we turn our bodies if we want to look behind ourselves for more than a brief glance).
To summarize, developers should carefully consider their control schemes and how to integrate head-tracking when designing applications for VR. The OculusRoomTiny application provides a source code sample that shows how to integrate Oculus head tracking with the aforementioned standard FPS control scheme.
For more information about good and bad practices, refer to the Oculus Best Practices Guide.
Health and Safety Warning
All applications that use the Oculus Rift periodically display a health and safety warning.
This warning appears for a short amount of time when the user wears the Rift; it can be dismissed by pressing a key or gazing at the acknowledgement. After the screen is dismissed, it shouldn't display for at least 30 minutes.
Oculus Rift | Rendering to the Oculus Rift | 13
Rendering to the Oculus Rift
The Oculus Rift requires split-screen stereo with distortion correction for each eye to cancel lens-related distortion.
Figure 3: OculusWorldDemo Stereo Rendering
Correcting for distortion can be challenging, with distortion parameters varying for different lens types and individual eye relief. To make development easier, Oculus SDK handles distortion correction automatically within the Oculus Compositor process; it also takes care of latency-reducing timewarp and presents frames to the headset.
With Oculus SDK doing a lot of the work, the main job of the application is to perform simulation and render stereo world based on the tracking pose. Stereo views can be rendered into either one or two individual textures and are submitted to the compositor by calling ovr_SubmitFrame. We cover this process in detail in this section.
Rendering to the Oculus Rift
The Oculus Rift requires the scene to be rendered in split-screen stereo with half of the screen used for each eye.
When using the Rift, the left eye sees the left half of the screen, and the right eye sees the right half.
Although varying from person-to-person, human eye pupils are approximately 65 mm apart. This is known as interpupillary distance (IPD). The in-application cameras should be configured with the same separation.
Note:
This is a translation of the camera, not a rotation, and it is this translation (and the parallax effect that goes with it) that causes the stereoscopic effect. This means that your application will need to render the entire scene twice, once with the left virtual camera, and once with the right.
The reprojection stereo rendering technique, which relies on left and right views being generated from a single fully rendered view, is usually not viable with an HMD because of significant artifacts at object edges.
14 | Rendering to the Oculus Rift | Oculus Rift
The lenses in the Rift magnify the image to provide a very wide field of view (FOV) that enhances immersion.
However, this process distorts the image significantly. If the engine were to display the original images on the
Rift, then the user would observe them with pincushion distortion.
Figure 4: Pincushion and Barrel Distortion
To counteract this distortion, the SDK applies post-processing to the rendered views with an equal and opposite barrel distortion so that the two cancel each other out, resulting in an undistorted view for each eye.
Furthermore, the SDK also corrects chromatic aberration, which is a color separation effect at the edges caused by the lens. Although the exact distortion parameters depend on the lens characteristics and eye position relative to the lens, the Oculus SDK takes care of all necessary calculations when generating the distortion mesh.
When rendering for the Rift, projection axes should be parallel to each other as illustrated in the following figure, and the left and right views are completely independent of one another. This means that camera setup is very similar to that used for normal non-stereo rendering, except that the cameras are shifted sideways to adjust for each eye location.
Figure 5: HMD Eye View Cones
In practice, the projections in the Rift are often slightly off-center because our noses get in the way! But the point remains, the left and right eye views in the Rift are entirely separate from each other, unlike stereo views generated by a television or a cinema screen. This means you should be very careful if trying to use methods developed for those media because they do not usually apply in VR.
The two virtual cameras in the scene should be positioned so that they are oriented in the same way as the eye poses, specified by ovrEyeRenderDesc::HmdToEyePose, and such that the distance between them is the same as the distance between the eyes, or the interpupillary distance (IPD).
Although the Rift’s lenses are approximately the right distance apart for most users, they may not exactly match the user’s IPD. However, because of the way the optics are designed, each eye will still see the correct view. It is important that the software makes the distance between the virtual cameras match the user’s IPD as found in their profile (set in the configuration utility), and not the distance between the Rift’s lenses.
Oculus Rift | Rendering to the Oculus Rift | 15
Rendering Setup Outline
The Oculus SDK makes use of a compositor process to present frames and handle distortion.
To target the Rift, you render the scene into one or two render textures, passing these textures into the API.
The Oculus runtime handles distortion rendering, GPU synchronization, frame timing, and frame presentation to the HMD.
The following are the steps for SDK rendering:
1. Initialize: a. Initialize Oculus SDK and create an ovrSession object for the headset as was described earlier.
b. Compute the desired FOV and texture sizes based on ovrHmdDesc data.
c. Allocate ovrTextureSwapChain objects, used to represent eye buffers, in an API-specific way: call ovr_CreateTextureSwapChainDX for either Direct3D 11 or 12, ovr_CreateTextureSwapChainGL for OpenGL, or ovr_CreateTextureSwapChainVk for Vulkan.
2. Set up frame handling: a. Use ovr_GetTrackingState and ovr_CalcEyePoses to compute eye poses needed for view rendering based on frame timing information.
b. Perform rendering for each eye in an engine-specific way, rendering into the current texture within the texture set. Current texture is retrieved using ovr_GetTextureSwapChainCurrentIndex and ovr_GetTextureSwapChainBufferDX, ovr_GetTextureSwapChainBufferGL, or ovr_GetTextureSwapChainBufferVk. After rendering to the texture is complete, the application must call ovr_CommitTextureSwapChain.
c. Call ovr_SubmitFrame, passing swap texture set(s) from the previous step within a ovrLayerEyeFov structure. Although a single layer is required to submit a frame, you can use multiple layers and layer types for advanced rendering. ovr_SubmitFrame passes layer textures to the compositor which handles distortion, timewarp, and GPU synchronization before presenting it to the headset.
3. Shutdown: a. Call ovr_DestroyTextureSwapChain to destroy swap texture buffers. Call ovr_DestroyMirrorTexture to destroy a mirror texture. To destroy the ovrSession object, call ovr_Destroy.
Texture Swap Chain Initialization
This section describes rendering initialization, including creation of texture swap chains.
Initially, you determine the rendering FOV and allocate the required ovrTextureSwapChain. The following code shows how the required texture size can be computed:
// Configure Stereo settings.
Sizei recommenedTex0Size = ovr_GetFovTextureSize(session, ovrEye_Left,
session->DefaultEyeFov[0], 1.0f);
Sizei recommenedTex1Size = ovr_GetFovTextureSize(session, ovrEye_Right,
session->DefaultEyeFov[1], 1.0f);
Sizei bufferSize; bufferSize.w = recommenedTex0Size.w + recommenedTex1Size.w; bufferSize.h = max ( recommenedTex0Size.h, recommenedTex1Size.h );
Render texture size is determined based on the FOV and the desired pixel density at the center of the eye. Although both the FOV and pixel density values can be modified to improve performance, this example uses the recommended FOV (obtained from session->DefaultEyeFov). The function ovr_GetFovTextureSize computes the desired texture size for each eye based on these parameters.
16 | Rendering to the Oculus Rift | Oculus Rift
The Oculus API allows the application to use either one shared texture or two separate textures for eye rendering. This example uses a single shared texture for simplicity, making it large enough to fit both eye renderings.
If you're using Vulkan, there are three steps that you need to add before you can create the texture swap chain.
• During initialization, call ovr_GetSessionPhysicalDeviceVk to get the current physical device matching the luid. Then, create a VkDevice associated with the returned physical device.
• AMD hardware uses differnet extensions on Vulkan. Add code similar to the following example to handle the AMD GPU extensions during app initalization. This code example comes from the
Win32_VulkanAppUtil.h that comes with the OculusRoomTiny_Advanced sample app that ships with the
Oculus SDK.
static const uint32_t AMDVendorId = 0x1002;
isAMD = (gpuProps.vendorID == AMDVendorId);
static const char* deviceExtensions[] =
{
VK_KHR_SWAPCHAIN_EXTENSION_NAME,
VK_KHX_EXTERNAL_MEMORY_EXTENSION_NAME,
#if defined(VK_USE_PLATFORM_WIN32_KHR)
VK_KHX_EXTERNAL_MEMORY_WIN32_EXTENSION_NAME,
#endif
};
static const char* deviceExtensionsAMD[] =
{
VK_KHR_SWAPCHAIN_EXTENSION_NAME
};
• Then, after the game loop has been established, identify which queue to synchronize when rendering. Call ovr_SetSynchonizationQueueVk to identify the queue.
Create the Texture Swap Chain
Once texture size is known, the application can call ovr_CreateTextureSwapChainGL, ovr_CreateTextureSwapChainDX, or ovr_CreateTextureSwapChainVk to allocate the texture swap chains in an API-specific way.
Here's how a texture swap chain can be created and accessed under OpenGL: ovrTextureSwapChain textureSwapChain = 0; ovrTextureSwapChainDesc desc = {}; desc.Type = ovrTexture_2D; desc.ArraySize = 1; desc.Format = OVR_FORMAT_R8G8B8A8_UNORM_SRGB; desc.Width = bufferSize.w; desc.Height = bufferSize.h; desc.MipLevels = 1; desc.SampleCount = 1; desc.StaticImage = ovrFalse; if (ovr_CreateTextureSwapChainGL(session, &desc, &textureSwapChain) == ovrSuccess)
{
// Sample texture access:
int texId;
ovr_GetTextureSwapChainBufferGL(session, textureSwapChain, 0, &texId);
glBindTexture(GL_TEXTURE_2D, texId);
...
}
Here's a similar example of texture swap chain creation and access using Direct3D 11: ovrTextureSwapChain textureSwapChain = 0; std::vector<ID3D11RenderTargetView*> texRtv; ovrTextureSwapChainDesc desc = {}; desc.Type = ovrTexture_2D;
Oculus Rift | Rendering to the Oculus Rift | 17
desc.Format = OVR_FORMAT_R8G8B8A8_UNORM_SRGB; desc.ArraySize = 1; desc.Width = bufferSize.w; desc.Height = bufferSize.h; desc.MipLevels = 1; desc.SampleCount = 1; desc.StaticImage = ovrFalse; desc.MiscFlags = ovrTextureMisc_None; desc.BindFlags = ovrTextureBind_DX_RenderTarget;
if (ovr_CreateTextureSwapChainDX(session, DIRECTX.Device, &desc, &textureSwapChain) == ovrSuccess)
{
int count = 0;
ovr_GetTextureSwapChainLength(session, textureSwapChain, &count);
texRtv.resize(textureCount);
for (int i = 0; i < count; ++i)
{
ID3D11Texture2D* texture = nullptr;
ovr_GetTextureSwapChainBufferDX(session, textureSwapChain, i, IID_PPV_ARGS(&texture));
DIRECTX.Device->CreateRenderTargetView(texture, nullptr, &texRtv[i]);
texture->Release();
}
}
Here's sample code from the provided OculusRoomTiny sample running in Direct3D 12: ovrTextureSwapChain TexChain; std::vector<D3D12_CPU_DESCRIPTOR_HANDLE> texRtv; std::vector<ID3D12Resource*> TexResource; ovrTextureSwapChainDesc desc = {}; desc.Type = ovrTexture_2D; desc.ArraySize = 1; desc.Format = OVR_FORMAT_R8G8B8A8_UNORM_SRGB; desc.Width = sizeW; desc.Height = sizeH; desc.MipLevels = 1; desc.SampleCount = 1; desc.MiscFlags = ovrTextureMisc_DX_Typeless; desc.StaticImage = ovrFalse; desc.BindFlags = ovrTextureBind_DX_RenderTarget;
// DIRECTX.CommandQueue is the ID3D12CommandQueue used to render the eye textures by the app ovrResult result = ovr_CreateTextureSwapChainDX(session, DIRECTX.CommandQueue, &desc, &TexChain); if (!OVR_SUCCESS(result))
return false; int textureCount = 0; ovr_GetTextureSwapChainLength(Session, TexChain, &textureCount); texRtv.resize(textureCount);
TexResource.resize(textureCount); for (int i = 0; i < textureCount; ++i)
{
result = ovr_GetTextureSwapChainBufferDX(Session, TexChain, i, IID_PPV_ARGS(&TexResource[i]));
if (!OVR_SUCCESS(result))
return false;
D3D12_RENDER_TARGET_VIEW_DESC rtvd = {};
rtvd.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
rtvd.ViewDimension = D3D12_RTV_DIMENSION_TEXTURE2D;
texRtv[i] = DIRECTX.RtvHandleProvider.AllocCpuHandle(); // Gives new D3D12_CPU_DESCRIPTOR_HANDLE
DIRECTX.Device->CreateRenderTargetView(TexResource[i], &rtvd, texRtv[i]);
}
Note: For Direct3D 12, when calling ovr_CreateTextureSwapChainDX, the caller provides a
ID3D12CommandQueue instead of a ID3D12Device to the SDK. It is the caller's responsibility to make sure that this ID3D12CommandQueue instance is where all VR eye-texture rendering is executed. Or, it can be used as a "join-node" fence to wait for the command lists executed by other command queues rendering the VR eye textures.
Here's how a texture swap chain can be created and accessed using Vulkan: bool Create(ovrSession aSession, VkExtent2D aSize, RenderPass& renderPass, DepthBuffer& depthBuffer)
{
18 | Rendering to the Oculus Rift | Oculus Rift
session = aSession;
size = aSize;
ovrTextureSwapChainDesc desc = {};
desc.Type = ovrTexture_2D;
desc.ArraySize = 1;
desc.Format = OVR_FORMAT_R8G8B8A8_UNORM_SRGB;
desc.Width = (int)size.width;
desc.Height = (int)size.height;
desc.MipLevels = 1;
desc.SampleCount = 1;
desc.MiscFlags = ovrTextureMisc_DX_Typeless;
desc.BindFlags = ovrTextureBind_DX_RenderTarget;
desc.StaticImage = ovrFalse;
ovrResult result = ovr_CreateTextureSwapChainVk(session, Platform.device, &desc, &textureChain);
if (!OVR_SUCCESS(result))
return false;
int textureCount = 0;
ovr_GetTextureSwapChainLength(session, textureChain, &textureCount);
texElements.reserve(textureCount);
for (int i = 0; i < textureCount; ++i)
{
VkImage image;
result = ovr_GetTextureSwapChainBufferVk(session, textureChain, i, &image);
texElements.emplace_back(RenderTexture());
CHECK(texElements.back().Create(image, VK_FORMAT_R8G8B8A8_SRGB, size, renderPass,
depthBuffer.view));
}
return true;
}
Once these textures and render targets are successfully created, you can use them to perform eye-texture rendering. The Frame Rendering section describes viewport setup in more detail.
The Oculus compositor provides sRGB-correct rendering, which results in more photorealistic visuals, better
MSAA, and energy-conserving texture sampling, which are very important for VR applications. As shown above, applications are expected to create sRGB texture swap chains. Proper treatment of sRGB rendering is a complex subject and, although this section provides an overview, extensive information is outside the scope of this document.
There are several steps to ensuring a real-time rendered application achieves sRGB-correct shading and different ways to achieve it. For example, most GPUs provide hardware acceleration to improve gamma-correct shading for sRGB-specific input and output surfaces, while some applications use GPU shader math for more customized control. For the Oculus SDK, when an application passes in sRGB-space texture swap chains, the compositor relies on the GPU's sampler to do the sRGB-to-linear conversion.
All color textures fed into a GPU shader should be marked appropriately with the sRGB-correct format, such as OVR_FORMAT_R8G8B8A8_UNORM_SRGB. This is also recommended for applications that provide static textures as quad-layer textures to the Oculus compositor. Failure to do so will cause the texture to look much brighter than expected.
For D3D 11 and 12, the texture format provided in desc for ovr_CreateTextureSwapChainDX is used by the distortion compositor for the ShaderResourceView when reading the contents of the texture.
As a result, the application should request texture swap chain formats that are in sRGB-space (e.g.
OVR_FORMAT_R8G8B8A8_UNORM_SRGB).
If your application is configured to render into a linear-format texture (e.g. OVR_FORMAT_R8G8B8A8_UNORM) and handles the linear-to-gamma conversion using HLSL code, or does not care about any gamma-correction, then:
• Request an sRGB format (e.g. OVR_FORMAT_R8G8B8A8_UNORM_SRGB) texture swap chain.
• Specify the ovrTextureMisc_DX_Typeless flag in the desc.
• Create a linear-format RenderTargetView (e.g. DXGI_FORMAT_R8G8B8A8_UNORM)
Oculus Rift | Rendering to the Oculus Rift | 19
Note: The ovrTextureMisc_DX_Typeless flag for depth buffer formats (e.g. OVR_FORMAT_D32) is ignored as they are always converted to be typeless.
The provided code sample demonstrates how to use the provided ovrTextureMisc_DX_Typeless flag in
D3D11:
ovrTextureSwapChainDesc desc = {};
desc.Type = ovrTexture_2D;
desc.ArraySize = 1;
desc.Format = OVR_FORMAT_R8G8B8A8_UNORM_SRGB;
desc.Width = sizeW;
desc.Height = sizeH;
desc.MipLevels = 1;
desc.SampleCount = 1;
desc.MiscFlags = ovrTextureMisc_DX_Typeless;
desc.BindFlags = ovrTextureBind_DX_RenderTarget;
desc.StaticImage = ovrFalse;
ovrResult result = ovr_CreateTextureSwapChainDX(session, DIRECTX.Device, &desc,
&textureSwapChain);
if(!OVR_SUCCESS(result))
return;
int count = 0;
ovr_GetTextureSwapChainLength(session, textureSwapChain, &count);
for (int i = 0; i < count; ++i)
{
ID3D11Texture2D* texture = nullptr;
ovr_GetTextureSwapChainBufferDX(session, textureSwapChain, i, IID_PPV_ARGS(&texture));
D3D11_RENDER_TARGET_VIEW_DESC rtvd = {};
rtvd.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
rtvd.ViewDimension = D3D11_RTV_DIMENSION_TEXTURE2D;
DIRECTX.Device->CreateRenderTargetView(texture, &rtvd, &texRtv[i]);
texture->Release();
}
For OpenGL, the format parameter ofovr_CreateTextureSwapChainGL is used by the distortion compositor when reading the contents of the texture. As a result, the application should request texture swap chain formats preferably in sRGB-space (e.g. OVR_FORMAT_R8G8B8A8_UNORM_SRGB). Furthermore, your application should call glEnable(GL_FRAMEBUFFER_SRGB); before rendering into these textures.
Even though it is not recommended, if your application is configured to treat the texture as a linear format (e.g.
GL_RGBA) and performs linear-to-gamma conversion in GLSL or does not care about gamma-correction, then:
• Request an sRGB format (e.g. OVR_FORMAT_R8G8B8A8_UNORM_SRGB) texture swap chain.
• Do not call glEnable(GL_FRAMEBUFFER_SRGB); when rendering into the texture.
For Vulkan, the format parameter of ovr_CreateTextureSwapChainVk is used by the distortion compositor when reading the contents of the texture. Your application should request texture swap chain formats in the sRGB-space (e.g. OVR_FORMAT_R8G8B8A8_UNORM_SRGB) as the compositor does sRGBcorrect rendering. The compositor will rely on the GPU’s hardware sampler to perform the sRGB-to-linear conversion.
If your application prefers rendering to a linear format (e.g. OVR_FORMAT_R8G8B8A8_UNORM) while handling the linear-to-gamma conversion via SPIRV code, the application must still request the corresponding sRGB format and also use ovrTextureMisc_DX_Typeless in the Flag field of ovrTextureSwapChainDesc. This allows the application to create a RenderTargetView in linear format, while allowing the compositor to treat it as sRGB. Failure to do this will result in unexpected gamma-curve artifacts. The ovrTextureMisc_DX_Typeless flag for depth buffer formats (e.g.
OVR_FORMAT_D32_FLOAT) is ignored as they are always converted to be typeless.
20 | Rendering to the Oculus Rift | Oculus Rift
In addition to sRGB, these concepts also apply to the mirror texture creation. For more information, refer to the function documentation provided for ovr_CreateMirrorTextureDX, ovr_CreateMirrorTextureGL, and ovr_CreateMirrorTextureWithOptionsVk for D3D, OpenGL, and Vulkan, respectively.
Frame Rendering
Frame rendering typically involves several steps: obtaining predicted eye poses based on the headset tracking pose, rendering the view for each eye and, finally, submitting eye textures to the compositor through ovr_SubmitFrame. After the frame is submitted, the Oculus compositor handles distortion and presents it on the Rift.
Before rendering frames it is helpful to initialize some data structures that can be shared across frames. As an example, we query eye descriptors and initialize the layer structure outside of the rendering loop:
// Initialize VR structures, filling out description.
ovrEyeRenderDesc eyeRenderDesc[2]; ovrPosef hmdToEyeViewPose[2]; ovrHmdDesc hmdDesc = ovr_GetHmdDesc(session); eyeRenderDesc[0] = ovr_GetRenderDesc(session, ovrEye_Left, hmdDesc.DefaultEyeFov[0]); eyeRenderDesc[1] = ovr_GetRenderDesc(session, ovrEye_Right, hmdDesc.DefaultEyeFov[1]); hmdToEyeViewPose[0] = eyeRenderDesc[0].HmdToEyePose; hmdToEyeViewPose[1] = eyeRenderDesc[1].HmdToEyePose;
// Initialize our single full screen Fov layer.
ovrLayerEyeFov layer; layer.Header.Type = ovrLayerType_EyeFov; layer.Header.Flags = 0; layer.ColorTexture[0] = textureSwapChain; layer.ColorTexture[1] = textureSwapChain; layer.Fov[0] = eyeRenderDesc[0].Fov; layer.Fov[1] = eyeRenderDesc[1].Fov; layer.Viewport[0] = Recti(0, 0, bufferSize.w / 2, bufferSize.h); layer.Viewport[1] = Recti(bufferSize.w / 2, 0, bufferSize.w / 2, bufferSize.h);
// ld.RenderPose and ld.SensorSampleTime are updated later per frame.
This code example first gets rendering descriptors for each eye, given the chosen FOV. The returned ovrEyeRenderDesc structure contains useful values for rendering, including the HmdToEyePose for each eye.
Eye view offsets are used later to adjust for eye separation.
The code also initializes the ovrLayerEyeFov structure for a full screen layer. Starting with Oculus SDK
0.6, frame submission uses layers to composite multiple view images or texture quads on top of each other.
This example uses a single layer to present a VR scene. For this purpose, we use ovrLayerEyeFov, which describes a dual-eye layer that covers the entire eye field of view. Since we are using the same texture set for both eyes, we initialize both eye color textures to pTextureSet and configure viewports to draw to the left and right sides of this shared texture, respectively.
Note: Although it is often enough to initialize viewports once in the beginning, specifying them as a part of the layer structure that is submitted every frame allows applications to change render target size dynamically, if desired. This is useful for optimizing rendering performance.
After setup completes, the application can run the rendering loop. First, we need to get the eye poses to render the left and right views.
// Get both eye poses simultaneously, with IPD offset already included.
double displayMidpointSeconds = GetPredictedDisplayTime(session, 0); ovrTrackingState hmdState = ovr_GetTrackingState(session, displayMidpointSeconds, ovrTrue); ovr_CalcEyePoses(hmdState.HeadPose.ThePose, hmdToEyeViewPose, layer.RenderPose);
In VR, rendered eye views depend on the headset position and orientation in the physical space, tracked with the help of internal IMU and external sensors. Prediction is used to compensate for the latency in the system, giving the best estimate for where the headset will be when the frame is displayed on the headset. In the
Oculus SDK, this tracked, predicted pose is reported by ovr_GetTrackingState.
Oculus Rift | Rendering to the Oculus Rift | 21
To do accurate prediction, ovr_GetTrackingState needs to know when the current frame will actually be displayed. The code above calls
GetPredictedDisplayTime to obtain displayMidpointSeconds for the current frame, using it to compute the best predicted tracking state. The head pose from the tracking state is then passed to ovr_CalcEyePoses to calculate correct view poses for each eye. These poses are stored directly into the layer.RenderPose[2] array. With eye poses ready, we can proceed onto the actual frame rendering.
if (isVisible)
{
// Get next available index of the texture swap chain
int currentIndex = 0;
ovr_GetTextureSwapChainCurrentIndex(session, textureSwapChain, ¤tIndex);
// Clear and set up render-target.
DIRECTX.SetAndClearRenderTarget(pTexRtv[currentIndex], pEyeDepthBuffer);
// Render Scene to Eye Buffers
for (int eye = 0; eye < 2; eye++)
{
// Get view and projection matrices for the Rift camera
Vector3f pos = originPos + originRot.Transform(layer.RenderPose[eye].Position);
Matrix4f rot = originRot * Matrix4f(layer.RenderPose[eye].Orientation);
Vector3f finalUp = rot.Transform(Vector3f(0, 1, 0));
Vector3f finalForward = rot.Transform(Vector3f(0, 0, -1));
Matrix4f view = Matrix4f::LookAtRH(pos, pos + finalForward, finalUp);
Matrix4f proj = ovrMatrix4f_Projection(layer.Fov[eye], 0.2f, 1000.0f, 0);
// Render the scene for this eye.
DIRECTX.SetViewport(layer.Viewport[eye]);
roomScene.Render(proj * view, 1, 1, 1, 1, true);
}
// Commit the changes to the texture swap chain
ovr_CommitTextureSwapChain(session, textureSwapChain);
}
// Submit frame with one layer we have.
ovrLayerHeader* layers = &layer.Header; ovrResult result = ovr_SubmitFrame(session, 0, nullptr, &layers, 1); isVisible = (result == ovrSuccess);
This code takes a number of steps to render the scene:
• It applies the texture as a render target and clears it for rendering. In this case, the same texture is used for both eyes.
• The code then computes view and projection matrices and sets viewport scene rendering for each eye.
In this example, view calculation combines the original pose ( originPos and originRot values) with the new pose computed based on the tracking state and stored in the layer. There original values can be modified by input to move the player within the 3D world.
• After texture rendering is complete, we call ovr_SubmitFrame to pass frame data to the compositor. From this point, the compositor takes over by accessing texture data through shared memory, distorting it, and presenting it on the Rift.
ovr_SubmitFrame returns once the submitted frame is queued up and the runtime is available to accept a new frame. When successful, its return value is either ovrSuccess or ovrSuccess_NotVisible.
ovrSuccess_NotVisible is returned if the frame wasn't actually displayed, which can happen when VR application loses focus. Our sample code handles this case by updating the isVisible flag, checked by the rendering logic. While frames are not visible, rendering should be paused to eliminate unnecessary GPU load.
If you receive ovrError_DisplayLost, the device was removed and the session is invalid. Release the shared resources (ovr_DestroyTextureSwapChain), destroy the session (ovr_Destroy), recreate it (ovr_Create), and create new resources (ovr_CreateTextureSwapChainXXX). The application's existing private graphics resources do not need to be recreated unless the new ovr_Create call returns a different
GraphicsLuid.
22 | Rendering to the Oculus Rift | Oculus Rift
Frame Timing
The Oculus SDK reports frame timing information through the ovr_GetPredictedDisplayTime function, relying on the application-provided frame index to ensure correct timing is reported across different threads.
Accurate frame and sensor timing are required for accurate head motion prediction, which is essential for a good VR experience. Prediction requires knowing exactly when in the future the current frame will appear on the screen. If we know both sensor and display scanout times, we can predict the future head pose and improve image stability. Computing these values incorrectly can lead to under or over-prediction, degrading perceived latency, and potentially causing overshoot “wobbles”.
To ensure accurate timing, the Oculus SDK uses absolute system time, stored as a double, to represent sensor and frame timing values. The current absolute time is returned by ovr_GetTimeInSeconds. Current time should rarely be used, however, since simulation and motion prediction will produce better results when relying on the timing values returned by ovr_GetPredictedDisplayTime. This function has the following signature: ovr_GetPredictedDisplayTime(ovrSession session, long long frameIndex);
The frameIndex argument specifies which application frame we are rendering. Applications that make use of multi-threaded rendering must keep an internal frame index and manually increment it, passing it across threads along with frame data to ensure correct timing and prediction. The same frameIndex value must be passed to ovr_SubmitFrame as was used to obtain timing for the frame. The details of multi-threaded timing are covered in the next section,
Rendering on Different Threads
A special frameIndex value of 0 can be used in both functions to request that the SDK keep track of frame indices automatically. However, this only works when all frame timing requests and render submission is done on the same thread.
Rendering on Different Threads
In some engines, render processing is distributed across more than one thread.
For example, one thread may perform culling and render setup for each object in the scene (we'll call this the
“main” thread), while a second thread makes the actual D3D or OpenGL API calls (we'll call this the “render” thread). Both of these threads may need accurate estimates of frame display time, so as to compute best possible predictions of head pose.
The asynchronous nature of this approach makes this challenging: while the render thread is rendering a frame, the main thread might be processing the next frame. This parallel frame processing may be out of sync by exactly one frame or a fraction of a frame, depending on game engine design. If we used the default global state to access frame timing, the result of
GetPredictedDisplayTime could either be off by one frame depending which thread the function is called from, or worse, could be randomly incorrect depending on how threads are scheduled. To address this issue, previous section introduced the concept of frameIndex that is tracked by the application and passed across threads along with frame data.
For multi-threaded rendering result to be correct, the following must be true: (a) pose prediction, computed based on frame timing, must be consistent for the same frame regardless of which thread it is accessed from; and (b) eye poses that were actually used for rendering must be passed into ovr_SubmitFrame, along with the frame index.
Here is a summary of steps you can take to ensure this is the case:
1. The main thread needs to assign a frame index to the current frame being processed for rendering. It would increment this index each frame and pass it to
GetPredictedDisplayTime to obtain the correct timing for pose prediction.
2. The main thread should call the thread safe function ovr_GetTrackingState with the predicted time value. It can also call ovr_CalcEyePoses if necessary for rendering setup.
Oculus Rift | Rendering to the Oculus Rift | 23
3. Main thread needs to pass the current frame index and eye poses to the render thread, along with any rendering commands or frame data it needs.
4. When the rendering commands executed on the render thread, developers need to make sure these things hold: a. The actual poses used for frame rendering are stored into the
RenderPose for the layer.
b. The same value of frameIndex as was used on the main thead is passed into ovr_SubmitFrame.
The following code illustrates this in more detail: void MainThreadProcessing()
{
frameIndex++;
// Ask the API for the times when this frame is expected to be displayed.
double frameTiming = GetPredictedDisplayTime(session, frameIndex);
// Get the corresponding predicted pose state.
ovrTrackingState state = ovr_GetTrackingState(session, frameTiming, ovrTrue);
ovrPosef eyePoses[2];
ovr_CalcEyePoses(state.HeadPose.ThePose, hmdToEyeViewOffset, eyePoses);
SetFrameHMDData(frameIndex, eyePoses);
// Do render pre-processing for this frame.
...
} void RenderThreadProcessing()
{
int frameIndex;
ovrPosef eyePoses[2];
GetFrameHMDData(&frameIndex, eyePoses);
layer.RenderPose[0] = eyePoses[0];
layer.RenderPose[1] = eyePoses[1];
// Execute actual rendering to eye textures.
...
// Submit frame with one layer we have.
ovrLayerHeader* layers = &layer.Header;
ovrResult result = ovr_SubmitFrame(session, frameIndex, nullptr, &layers, 1);
}
The Oculus SDK also supports Direct3D 12, which allows submitting rendering work to the GPU from multiple
CPU threads. When the application calls ovr_CreateTextureSwapChainDX, the Oculus SDK caches off the
ID3D12CommandQueue provided by the caller for future usage. As the application calls ovr_SubmitFrame, the SDK drops a fence on the cached ID3D12CommandQueue to know exactly when a given set of eyetextures are ready for the SDK compositor.
For a given application, using a single ID3D12CommandQueue on a single thread is the easiest. But,it might also split the CPU rendering workload for each eye-texture pair or push non-eye-texture rendering work, such as shadows, reflection maps, and so on, onto different command queues. If the application populates and executes command lists from multiple threads, it will also have to make sure that the ID3D12CommandQueue provided to the SDK is the single join-node for the eye-texture rendering work executed through different command queues.
Layers
Similar to the way a monitor view can be composed of multiple windows, the display on the headset can be composed of multiple layers. Typically at least one of these layers will be a view rendered from the user's virtual eyeballs, but other layers may be HUD layers, information panels, text labels attached to items in the world, aiming reticles, and so on.
24 | Rendering to the Oculus Rift | Oculus Rift
Each layer can have a different resolution, can use a different texture format, can use a different field of view or size, and might be in mono or stereo. The application can also be configured to not update a layer's texture if the information in it has not changed. For example, it might not update if the text in an information panel has not changed since last frame or if the layer is a picture-in-picture view of a video stream with a low framerate.
Applications can supply mipmapped textures to a layer and, together with a high-quality distortion mode, this is very effective at improving the readability of text panels.
Every frame, all active layers are composited from back to front using pre-multiplied alpha blending. Layer 0 is the furthest layer, layer 1 is on top of it, and so on; there is no depth-buffer intersection testing of layers, even if a depth-buffer is supplied.
A powerful feature of layers is that each can be a different resolution. This allows an application to scale to lower performance systems by dropping resolution on the main eye-buffer render that shows the virtual world, but keeping essential information, such as text or a map, in a different layer at a higher resolution.
There are several layer types available:
EyeFov
Quad
EyeMatrix
Disabled
The standard "eye buffer" familiar from previous SDKs, which is typically a stereo view of a virtual scene rendered from the position of the user's eyes. Although eye buffers can be mono, this can cause discomfort. Previous SDKs had an implicit field of view (FOV) and viewport; these are now supplied explicitly and the application can change them every frame, if desired.
A monoscopic image that is displayed as a rectangle at a given pose and size in the virtual world. This is useful for heads-up-displays, text information, object labels and so on. By default the pose is specified relative to the user's real-world space and the quad will remain fixed in space rather than moving with the user's head or body motion. For head-locked quads, use the ovrLayerFlag_HeadLocked flag as described below.
The EyeMatrix layer type is similar to the EyeFov layer type and is provided to assist compatibility with Gear VR applications. For more information, refer to the
Mobile SDK documentation.
Ignored by the compositor, disabled layers do not cost performance. We recommend that applications perform basic frustum-culling and disable layers that are out of view. However, there is no need for the application to repack the list of active layers tightly together when turning one layer off; disabling it and leaving it in the list is sufficient. Equivalently, the pointer to the layer in the list can be set to null.
Each layer style has a corresponding member of the ovrLayerType enum, and an associated structure holding the data required to display that layer. For example, the EyeFov layer is type number ovrLayerType_EyeFov and is described by the data in the structure ovrLayerEyeFov. These structures share a similar set of parameters, though not all layer types require all parameters:
Parameter Type Description
Header.Type
Header.Flags
ColorTexture enum ovrLayerType Must be set by all layers to specify what type they are.
A bitfield of ovrLayerFlags
See below for more information.
TextureSwapChain
Provides color and translucency data for the layer. Layers are blended over one another using premultiplied alpha. This allows them to express either lerp-style blending, additive blending, or a combination of the two. Layer textures must be RGBA or BGRA formats and might have mipmaps, but cannot be arrays, cubes, or have MSAA. If the application
Parameter
Viewport
Fov
RenderPose
SensorSampleTime
QuadPoseCenter
QuadSize
Type ovrRecti ovrFovPort ovrPosef double ovrPosef ovrVector2f
Oculus Rift | Rendering to the Oculus Rift | 25
Description desires to do MSAA rendering, then it must resolve the intermediate MSAA color texture into the layer's non-MSAA
ColorTexture.
The rectangle of the texture that is actually used, specified in 0-1 texture "UV" coordinate space (not pixels). In theory, texture data outside this region is not visible in the layer.
However, the usual caveats about texture sampling apply, especially with mipmapped textures. It is good practice to leave a border of RGBA(0,0,0,0) pixels around the displayed region to avoid "bleeding," especially between two eye buffers packed side by side into the same texture. The size of the border depends on the exact usage case, but around
8 pixels seems to work well in most cases.
The field of view used to render the scene in an Eye layer type. Note this does not control the HMD's display, it simply tells the compositor what FOV was used to render the texture data in the layer - the compositor will then adjust appropriately to whatever the actual user's FOV is. Applications may change FOV dynamically for special effects. Reducing FOV may also help with performance on slower machines, though typically it is more effective to reduce resolution before reducing FOV.
The camera pose the application used to render the scene in an Eye layer type. This is typically predicted by the SDK and application using the ovr_GetTrackingState and ovr_CalcEyePoses functions. The difference between this pose and the actual pose of the eye at display time is used by the compositor to apply timewarp to the layer.
The absolute time when the application sampled the tracking state. The typical way to acquire this value is to have an ovr_GetTimeInSeconds call right next to the ovr_GetTrackingState call. The SDK uses this value to report the application's motion-to-photon latency in the
Performance HUD. If the application has more than one ovrLayerType_EyeFov layer submitted at any given frame, the SDK scrubs through those layers and selects the timing with the lowest latency. In a given frame, if no ovrLayerType_EyeFov layers are submitted, the SDK will use the point in time when ovr_GetTrackingState was called with the latencyMarkerset to ovrTrue as the substitute application motion-to-photon latency time.
Specifies the orientation and position of the center point of a Quad layer type. The supplied direction is the vector perpendicular to the quad. The position is in real-world meters (not the application's virtual world, the actual world the user is in) and is relative to the
"zero" position set by ovr_RecenterTrackingOrigin or ovr_SpecifyTrackingOrigin unless the ovrLayerFlag_HeadLocked flag is used.
Specifies the width and height of a Quad layer type. As with position, this is in real-world meters.
26 | Rendering to the Oculus Rift | Oculus Rift
Layers that take stereo information (all those except Quad layer types) take two sets of most parameters, and these can be used in three different ways:
• Stereo data, separate textures—the app supplies a different ovrTextureSwapChain for the left and right eyes, and a viewport for each.
• Stereo data, shared texture—the app supplies the same ovrTextureSwapChain for both left and right eyes, but a different viewport for each. This allows the application to render both left and right views to the same texture buffer. Remember to add a small buffer between the two views to prevent "bleeding", as discussed above.
• Mono data—the app supplies the same ovrTextureSwapChain for both left and right eyes, and the same viewport for each.
Texture and viewport sizes may be different for the left and right eyes, and each can even have different fields of view. However beware of causing stereo disparity and discomfort in your users.
The
Header.Flags field available for all layers is a logical-or of the following:
• ovrLayerFlag_HighQuality—enables 4x anisotropic sampling in the compositor for this layer. This can provide a significant increase in legibility, especially when used with a texture containing mipmaps; this is recommended for high-frequency images such as text or diagrams and when used with the Quad layer types. For Eye layer types, it will also increase visual fidelity towards the periphery, or when feeding in textures that have more than the 1:1 recommended pixel density. For best results, when creating mipmaps for the textures associated to the particular layer, make sure the texture sizes are a power of 2. However, the application does not need to render to the whole texture; a viewport that renders to the recommended size in the texture will provide the best performance-to-quality ratios.
• ovrLayerFlag_TextureOriginAtBottomLeft—the origin of a layer's texture is assumed to be at the top-left corner. However, some engines (particularly those using OpenGL) prefer to use the bottom-left corner as the origin, and they should use this flag.
• ovrLayerFlag_HeadLocked—Most layer types have their pose orientation and position specified relative to the "zero position" defined by calling ovr_RecenterTrackingOrigin. However the app may wish to specify a layer's pose relative to the user's face. When the user moves their head, the layer follows. This is useful for reticles used in gaze-based aiming or selection. This flag may be used for all layer types, though it has no effect when used on the Direct type.
At the end of each frame, after rendering to whichever ovrTextureSwapChain the application wants to update and calling ovr_CommitTextureSwapChain, the data for each layer is put into the relevant ovrLayerEyeFov / ovrLayerQuad / ovrLayerDirect structure. The application then creates a list of pointers to those layer structures, specifically to the
Header field which is guaranteed to be the first member of each structure. Then the application builds a ovrViewScaleDesc struct with the required data, and calls the ovr_SubmitFrame function.
// Create eye layer.
ovrLayerEyeFov eyeLayer; eyeLayer.Header.Type = ovrLayerType_EyeFov; eyeLayer.Header.Flags = 0; for ( int eye = 0; eye < 2; eye++ )
{
eyeLayer.ColorTexture[eye] = EyeBufferSet[eye];
eyeLayer.Viewport[eye] = EyeViewport[eye];
eyeLayer.Fov[eye] = EyeFov[eye];
eyeLayer.RenderPose[eye] = EyePose[eye];
}
// Create HUD layer, fixed to the player's torso ovrLayerQuad hudLayer; hudLayer.Header.Type = ovrLayerType_Quad; hudLayer.Header.Flags = ovrLayerFlag_HighQuality; hudLayer.ColorTexture = TheHudTextureSwapChain;
// 50cm in front and 20cm down from the player's nose,
// fixed relative to their torso.
hudLayer.QuadPoseCenter.Position.x = 0.00f; hudLayer.QuadPoseCenter.Position.y = -0.20f; hudLayer.QuadPoseCenter.Position.z = -0.50f;
Oculus Rift | Rendering to the Oculus Rift | 27
hudLayer.QuadPoseCenter.Orientation.x = 0; hudLayer.QuadPoseCenter.Orientation.y = 0; hudLayer.QuadPoseCenter.Orientation.z = 0; hudLayer.QuadPoseCenter.Orientation.w = 1;
// HUD is 50cm wide, 30cm tall.
hudLayer.QuadSize.x = 0.50f; hudLayer.QuadSize.y = 0.30f;
// Display all of the HUD texture.
hudLayer.Viewport.Pos.x = 0.0f; hudLayer.Viewport.Pos.y = 0.0f; hudLayer.Viewport.Size.w = 1.0f; hudLayer.Viewport.Size.h = 1.0f;
// The list of layers.
ovrLayerHeader *layerList[2]; layerList[0] = &eyeLayer.Header; layerList[1] = &hudLayer.Header;
// Set up positional data.
ovrViewScaleDesc viewScaleDesc; viewScaleDesc.HmdSpaceToWorldScaleInMeters = 1.0f; viewScaleDesc.HmdToEyeViewOffset[0] = HmdToEyePose[0]; viewScaleDesc.HmdToEyeViewOffset[1] = HmdToEyePose[1]; ovrResult result = ovr_SubmitFrame(Session, 0, &viewScaleDesc, layerList, 2);
The compositor performs timewarp, distortion, and chromatic aberration correction on each layer separately before blending them together. The traditional method of rendering a quad to the eye buffer involves two filtering steps (once to the eye buffer, then once during distortion). Using layers, there is only a single filtering step between the layer image and the final framebuffer. This can provide a substantial improvement in text quality, especially when combined with mipmaps and the ovrLayerFlag_HighQuality flag.
One current disadvantage of layers is that no post-processing can be performed on the final composited image, such as soft-focus effects, light-bloom effects, or the Z intersection of layer data. Some of these effects can be performed on the contents of the layer with similar visual results.
Calling ovr_SubmitFrame queues the layers for display, and transfers control of the committed textures inside the ovrTextureSwapChains to the compositor. It is important to understand that these textures are being shared (rather than copied) between the application and the compositor threads, and that composition does not necessarily happen at the time ovr_SubmitFrame is called, so care must be taken. To continue rendering into a texture swap chain the application should always get the next available index with ovr_GetTextureSwapChainCurrentIndex before rendering into it. For example:
// Create two TextureSwapChains to illustrate.
ovrTextureSwapChain eyeTextureSwapChain; ovr_CreateTextureSwapChainDX ( ... &eyeTextureSwapChain ); ovrTextureSwapChain hudTextureSwapChain; ovr_CreateTextureSwapChainDX ( ... &hudTextureSwapChain );
// Set up two layers.
ovrLayerEyeFov eyeLayer; ovrLayerEyeFov hudLayer; eyeLayer.Header.Type = ovrLayerType_EyeFov; eyeLayer...etc... // set up the rest of the data.
hudLayer.Header.Type = ovrLayerType_Quad; hudLayer...etc... // set up the rest of the data.
// the list of layers ovrLayerHeader *layerList[2]; layerList[0] = &eyeLayer.Header; layerList[1] = &hudLayer.Header;
// Each frame...
int currentIndex = 0; ovr_GetTextureSwapChainCurrentIndex(... eyeTextureSwapChain, ¤tIndex);
// Render into it. It is recommended the app use ovr_GetTextureSwapChainBufferDX for each index on
texture chain creation to cache
// textures or create matching render target views. Each frame, the currentIndex value returned can
be used to index directly into that.
ovr_CommitTextureSwapChain(... eyeTextureSwapChain); ovr_GetTextureSwapChainCurrentIndex(... hudTextureSwapChain, ¤tIndex);
28 | Rendering to the Oculus Rift | Oculus Rift
// Render into it. It is recommended the app use ovr_GetTextureSwapChainBufferDX for each index on
texture chain creation to cache
// textures or create matching render target views. Each frame, the currentIndex value returned can
be used to index directly into that.
ovr_CommitTextureSwapChain(... hudTextureSwapChain); eyeLayer.ColorTexture[0] = eyeTextureSwapChain; eyeLayer.ColorTexture[1] = eyeTextureSwapChain; hudLayer.ColorTexture = hudTextureSwapChain; ovr_SubmitFrame(Hmd, 0, nullptr, layerList, 2);
Working with HMD Eye Poses
In the Oculus PC SDK, prior to version 1.17, eye poses only had three degrees-of-freedom (DOF), i.e. only translation. Eye poses were specified in the
HmdToEyeOffsetvector provided by the ovr_GetRenderDesc function. Starting with version 1.17,
HmdToEyeOffset has been renamed to HmdToEyePose using the type ovrPosef which contains a Position and Orientation, effectively giving eye poses six degrees-offreedom. This means that each eye’s render frustum can now be rotated away from the HMD’s orientation, in addition to being translated by the SDK. Because of this, the eye frustums’ axes are no longer guaranteed to be parallel to each other or to the HMD’s orientation axes. This generalization provides greater freedom to the SDK in defining the HMD geometry. But it also means that, as a VR app developer, you need to be more careful about your previous assumptions, especially when it comes to rendering.
Here are some pointers to make sure your VR app is correctly using
HmdToEyePose:
• If your VR app needs the vector translation value of the (pre-version 1.17)
HmdToEyeOffset, you can use
HmdToEyePose.Position instead. However, unless you are absolutely sure about what you are doing, there is a good chance you actually want to treat
HmdToEyePose as a whole transform, rather than separate out
Orientation from Position.
• Prior to PC-SDK version 1.17, rendering a 2D quad flat across the screen (e.g. a rectangle) would have always been acceptable. But with the possibility of rotating each eye frustum independently, your VR app will need to incorporate each eye’s orientation into the transformation of the quad so that the quad is rendered with the correct perspective in each eye. Here is a (somewhat exaggerated) example:
Oculus Rift | Rendering to the Oculus Rift | 29
The idea is to orient the quad such that it appears to use either the “center-eye” orientation, or the HMD orientation. This also applies to other screen-aligned quads, such as 3D splash screens, or particle effects such as large flip-book smoke quads. Except for particles, avoid rendering such quads natively; prefer using ovrLayerQuad instead.
• Some VR apps generate a single monoscopic camera frustum from the ovrFovPort structures of both eyes, in order to take advantage of various rendering optimizations. This is normally done by using an ovrFovPort, which takes the maximum of the ovrFovPort values for both eyes, on all four sides of the frustum. Before generating the monoscopic frustum this way however, be sure to remove any potential rotation from the ovrFovPort values by calling FovPort::Uncant, which is located in the ovr_math.h
header. See the
OculusWorldDemo sample code, to see how to use FovPort::Uncant.
Asynchronous TimeWarp
Asynchronous TimeWarp (ATW) is a technique for reducing latency and judder in VR applications and experiences.
In a basic VR game loop, the following occurs:
1. The software requests your head position.
2. The CPU processes the scene for each eye.
3. The GPU renders the scenes.
30 | Rendering to the Oculus Rift | Oculus Rift
4. The Oculus Compositor applies distortion and displays the scenes on the headset.
The following shows a basic example of a game loop:
Figure 6: Basic Game Loop
When frame rate is maintained, the experience feels real and is enjoyable. When it doesn’t happen in time, the previous frame is shown which can be disorienting. The following graphic shows an example of judder during the basic game loop:
Figure 7: Basic Game Loop with Judder
When you move your head and the world doesn’t keep up, this can be jarring and break immersion.
ATW is a technique that shifts the rendered image slightly to adjust for changes in head movement. Although the image is modified, your head does not move much, so the change is slight.
Additionally, to smooth issues with the user’s computer, game design or the operating system, ATW can help fix “potholes” or moments when the frame rate unexpectedly drops.
The following graphic shows an example of frame drops when ATW is applied:
Figure 8: Game Loop with ATW
At the refresh interval, the Compositor applies TimeWarp to the last rendered frame. As a result, a TimeWarped frame will always be shown to the user, regardless of frame rate. If the frame rate is very bad, flicker will be noticeable at the periphery of the display. But, the image will still be stable.
ATW is automatically applied by the Oculus Compositor; you do not need to enable or tune it. However, although ATW reduces latency, make sure that your application or experience makes frame rate.
Adaptive Queue Ahead
To improve CPU and GPU parallelism and increase the amount of time that the GPU has to process a frame, the
SDK automatically applies queue ahead up to 1 frame.
Oculus Rift | Rendering to the Oculus Rift | 31
Without queue ahead, the CPU begins processing the next frame immediately after the previous frame displays. After the CPU finishes, the GPU processes the frame, the compositor applies distortion, and the frame is displayed to the user. The following graphic shows CPU and GPU utilization without queue ahead:
Figure 9: CPU and GPU Utilization without Queue Ahead
If the GPU cannot process the frame in time for display, the previous frame displays. This results in judder.
With queue ahead, the CPU can start earlier; this provides the GPU more time to process the frame. The following graphic shows CPU and GPU utilization with queue ahead:
Figure 10: CPU and GPU Utilization with Queue Ahead
32 | Advanced Rendering Configuration | Oculus Rift
Advanced Rendering Configuration
By default, the SDK generates configuration values that optimize for rendering quality.
It also provides a degree of flexibility. For example, you can make changes when creating render target textures.
This section discusses changes you can make when choosing between rendering quality and performance, or if the engine you are using imposes constraints.
Coping with Graphics API or Hardware
Rendertarget Granularity
The SDK is designed with the assumption that you want to use your video memory as carefully as possible and that you can create exactly the right render target size for your needs.
However, real video cards and real graphics APIs have size limitations (all have a maximum size; some also have a minimum size). They might also have granularity restrictions, for example, only being able to create render targets that are a multiple of 32 pixels in size or having a limit on possible aspect ratios. As an application developer, you can also impose extra restrictions to avoid using too much graphics memory.
In addition to the above, the size of the actual render target surface in memory might not necessarily be the same size as the portion that is rendered to. The latter may be slightly smaller. However, since it is specified as a viewport, it typically does not have any granularity restrictions. When you bind the render target as a texture, however, it is the full surface that is used, and so the UV coordinates must be corrected for the difference between the size of the rendering and the size of the surface it is on. The API will do this for you, but you need to tell it the relevant information.
The following code shows a two-stage approach for settings render target resolution. The code first calls ovr_GetFovTextureSize to compute the ideal size of the render target. Next, the graphics library is called to create a render target of the desired resolution. In general, due to idiosyncrasies of the platform and hardware, the resulting texture size might be different from that requested.
// Get recommended left and right eye render target sizes.
Sizei recommenedTex0Size = ovr_GetFovTextureSize(session, ovrEye_Left,
session->DefaultEyeFov[0], pixelsPerDisplayPixel);
Sizei recommenedTex1Size = ovr_GetFovTextureSize(session, ovrEye_Right,
session->DefaultEyeFov[1], pixelsPerDisplayPixel);
// Determine dimensions to fit into a single render target.
Sizei renderTargetSize;
renderTargetSize.w = recommenedTex0Size.w + recommenedTex1Size.w;
renderTargetSize.h = max ( recommenedTex0Size.h, recommenedTex1Size.h );
// Create texture.
pRendertargetTexture = pRender->CreateTexture(renderTargetSize.w, renderTargetSize.h);
// The actual RT size may be different due to HW limits.
renderTargetSize.w = pRendertargetTexture->GetWidth();
renderTargetSize.h = pRendertargetTexture->GetHeight();
// Initialize eye rendering information.
// The viewport sizes are re-computed in case RenderTargetSize changed due to HW limitations.
ovrFovPort eyeFov[2] = { session->DefaultEyeFov[0], session->DefaultEyeFov[1] };
EyeRenderViewport[0].Pos = Vector2i(0,0);
EyeRenderViewport[0].Size = Sizei(renderTargetSize.w / 2, renderTargetSize.h);
EyeRenderViewport[1].Pos = Vector2i((renderTargetSize.w + 1) / 2, 0);
EyeRenderViewport[1].Size = EyeRenderViewport[0].Size;
Oculus Rift | Advanced Rendering Configuration | 33
This data is passed into ovr_SubmitFrame as part of the layer description.
You are free to choose the render target texture size and left and right eye viewports as you like, provided that you specify these values when calling ovr_SubmitFrame using the ovrTexture. However, using ovr_GetFovTextureSize will ensure that you allocate the optimum size for the particular HMD in use. The following sections describe how to modify the default configurations to make quality and performance tradeoffs. You should also note that the API supports using different render targets for each eye if that is required by your engine (although using a single render target is likely to perform better since it will reduce context switches). OculusWorldDemo allows you to toggle between using a single combined render target versus separate ones for each eye, by navigating to the settings menu (press the Tab key) and selecting the Share
RenderTarget option.
Forcing a Symmetrical Field of View
Typically the API will return an FOV for each eye that is not symmetrical, meaning the left edge is not the same distance from the center as the right edge.
This is because humans, as well as the Rift, have a wider FOV when looking outwards. When you look inwards, your nose is in the way. We are also better at looking down than we are at looking up. For similar reasons, the
Rift’s view is not symmetrical. It is controlled by the shape of the lens, various bits of plastic, and the edges of the screen. The exact details depend on the shape of your face, your IPD, and where precisely you place the
Rift on your face; all of this is set up in the configuration tool and stored in the user profile. All of this means that almost nobody has all four edges of their FOV set to the same angle, so the frustum produced will be offcenter. In addition, most people will not have the same fields of view for both their eyes. They will be close, but rarely identical.
As an example, on our first generation DK1 headset, the author’s left eye has the following FOV:
• 53.6 degrees up
• 58.9 degrees down
• 50.3 degrees inwards (towards the nose)
• 58.7 degrees outwards (away from the nose)
In the code and documentation, these are referred to as ‘half angles’ because traditionally a FOV is expressed as the total edge-to-edge angle. In this example, the total horizontal FOV is 50.3+58.7 = 109.0 degrees, and the total vertical FOV is 53.6+58.9 = 112.5 degrees.
The recommended and maximum fields of view can be accessed from the HMD as shown below: ovrFovPort defaultLeftFOV = session->DefaultEyeFov[ovrEye_Left]; ovrFovPort maxLeftFOV = session->MaxEyeFov[ovrEye_Left];
DefaultEyeFov refers to the recommended FOV values based on the current user’s profile settings (IPD, eye relief etc).
MaxEyeFov refers to the maximum FOV that the headset can possibly display, regardless of profile settings.
The default values provide a good user experience with no unnecessary additional GPU load. If your application does not consume significant GPU resources, you might want to use the maximum FOV settings to reduce reliance on the accuracy of the profile settings. You might provide a slider in the application control panel that enables users to choose interpolated FOV settings between the default and the maximum. But, if your application is heavy on GPU usage, you might want to reduce the FOV below the default values as described in
Improving Performance by Decreasing Field of View
34 | Advanced Rendering Configuration | Oculus Rift
The FOV angles for up, down, left, and right (expressed as the tangents of the half-angles), is the most convenient form to set up culling or portal boundaries in your graphics engine. The FOV values are also used to determine the projection matrix used during left and right eye scene rendering. We provide an API utility function ovrMatrix4f_Projection for this purpose: ovrFovPort fov;
// Determine fov.
...
ovrMatrix4f projMatrix = ovrMatrix4f_Projection(fov, znear, zfar, 0);
It is common for the top and bottom edges of the FOV to not be the same as the left and right edges when viewing a PC monitor. This is commonly called the ‘aspect ratio’ of the display, and very few displays are square. However, some graphics engines do not support off-center frustums. To be compatible with these engines, you will need to modify the FOV values reported by the ovrHmdDesc struct. In general, it is better to grow the edges than to shrink them. This will put a little more strain on the graphics engine, but will give the user the full immersive experience, even if they won’t be able to see some of the pixels being rendered.
Some graphics engines require that you express symmetrical horizontal and vertical fields of view, and some need an even less direct method such as a horizontal FOV and an aspect ratio. Some also object to having frequent changes of FOV, and may insist that both eyes be set to the same. The following is a an example of code for handling this restrictive case: ovrFovPort fovLeft = session->DefaultEyeFov[ovrEye_Left]; ovrFovPort fovRight = session->DefaultEyeFov[ovrEye_Right]; ovrFovPort fovMax = FovPort::Max(fovLeft, fovRight); float combinedTanHalfFovHorizontal = max ( fovMax.LeftTan, fovMax.RightTan ); float combinedTanHalfFovVertical = max ( fovMax.UpTan, fovMax.DownTan ); ovrFovPort fovBoth; fovBoth.LeftTan = fovBoth.RightTan = combinedTanHalfFovHorizontal; fovBoth.UpTan = fovBoth.DownTan = combinedTanHalfFovVertical;
// Create render target.
Sizei recommenedTex0Size = ovr_GetFovTextureSize(session, ovrEye_Left,
fovBoth, pixelsPerDisplayPixel);
Sizei recommenedTex1Size = ovr_GetFovTextureSize(session, ovrEye_Right,
fovBoth, pixelsPerDisplayPixel);
...
// Initialize rendering info.
ovrFovPort eyeFov[2]; eyeFov[0] = fovBoth; eyeFov[1] = fovBoth;
...
// Compute the parameters to feed to the rendering engine.
// In this case we are assuming it wants a horizontal FOV and an aspect ratio.
float horizontalFullFovInRadians = 2.0f * atanf ( combinedTanHalfFovHorizontal ); float aspectRatio = combinedTanHalfFovHorizontal / combinedTanHalfFovVertical;
GraphicsEngineSetFovAndAspect ( horizontalFullFovInRadians, aspectRatio );
...
Note: You will need to determine FOV before creating the render targets, since FOV affects the size of the recommended render target required for a given quality.
Oculus Rift | Advanced Rendering Configuration | 35
Improving Performance by Decreasing Pixel Density
The DK1 has a resolution of 1280x800 pixels, split between the two eyes. However, because of the wide FOV of the Rift and the way perspective projection works, the size of the intermediate render target required to match the native resolution in the center of the display is significantly higher.
For example, to achieve a 1:1 pixel mapping in the center of the screen for the author’s field-of-view settings on a DK1 requires a much larger render target that is 2000x1056 pixels in size.
Even if modern graphics cards can render this resolution at the required 60Hz, future HMDs might have significantly higher resolutions. For virtual reality, dropping below 60Hz provides a terrible user experience; it is always better to decrease the resolution to maintain framerate. This is similar to a user having a high resolution
2560x1600 monitor. Very few 3D applications can run at this native resolution at full speed, so most allow the user to select a lower resolution to which the monitor upscales to the fill the screen.
You can use the same strategy on the HMD. That is, run it at a lower video resolution and let the hardware upscale for you. However, this introduces two steps of filtering: one by the distortion processing and one by the video upscaler. Unfortunately, this double filtering introduces significant artifacts. It is usually more effective to leave the video mode at the native resolution, but limit the size of the intermediate render target. This gives a similar increase in performance, but preserves more detail.
One way to resolve this is to allow the user to adjust the resolution through a resolution selector. However, the actual resolution of the render target depends on the user’s configuration, rather than a standard hardware setting This means that the ‘native’ resolution is different for different people. Additionally, presenting resolutions higher than the physical hardware resolution might confuse some users. They might not understand that selecting 1280x800 is a significant drop in quality, even though this is the resolution reported by the hardware.
A better option is to modify the pixelsPerDisplayPixel value that is passed to the ovr_GetFovTextureSize function. This could also be based on a slider presented in the applications render settings. This determines the relative size of render target pixels as they map to pixels at the center of the display surface. For example, a value of 0.5 would reduce the render target size from 2000x1056 to 1000x528 pixels, which might allow mid-range PC graphics cards to maintain 60Hz.
float pixelsPerDisplayPixel = GetPixelsPerDisplayFromApplicationSettings();
Sizei recommenedTexSize = ovr_GetFovTextureSize(session, ovrEye_Left, fovLeft,
pixelsPerDisplayPixel);
Although you can set the parameter to a value larger than 1.0 to produce a higher-resolution intermediate render target, Oculus hasn't observed any useful increase in quality and it has a high performance cost.
OculusWorldDemo allows you to experiment with changing the render target pixel density. Navigate to the settings menu (press the Tab key) and select Pixel Density. Press the up and down arrow keys to adjust the pixel density at the center of the eye projection. A value of 1.0 sets the render target pixel density to the display surface 1:1 at this point on the display. A value of 0.5 means sets the density of the render target pixels to half of the display surface. Additionally, you can select Dynamic Res Scaling which will cause the pixel density to automatically adjust between 0 to 1.
36 | Advanced Rendering Configuration | Oculus Rift
Improving Performance by Decreasing Field of View
In addition to reducing the number of pixels in the intermediate render target, you can increase performance by decreasing the FOV that the pixels are stretched across.
Depending on the reduction, this can result in tunnel vision which decreases the sense of immersion.
Nevertheless, reducing the FOV increases performance in two ways. The most obvious is fillrate. For a fixed pixel density on the retina, a lower FOV has fewer pixels. Because of the properties of projective math, the outermost edges of the FOV are the most expensive in terms of numbers of pixels. The second reason is that there are fewer objects visible in each frame which implies less animation, fewer state changes, and fewer draw calls.
Reducing the FOV set by the player is a very painful choice to make. One of the key experiences of virtual reality is being immersed in the simulated world, and a large part of that is the wide FOV. Losing that aspect is not a thing we would ever recommend happily. However, if you have already sacrificed as much resolution as you can, and the application is still not running at 60Hz on the user’s machine, this is an option of last resort.
We recommend giving players a Maximum FOV slider that defines the four edges of each eye’s FOV.
ovrFovPort defaultFovLeft = session->DefaultEyeFov[ovrEye_Left]; ovrFovPort defaultFovRight = session->DefaultEyeFov[ovrEye_Right]; float maxFovAngle = ...get value from game settings panel...; float maxTanHalfFovAngle = tanf ( DegreeToRad ( 0.5f * maxFovAngle ) ); ovrFovPort newFovLeft = FovPort::Min(defaultFovLeft, FovPort(maxTanHalfFovAngle)); ovrFovPort newFovRight = FovPort::Min(defaultFovRight, FovPort(maxTanHalfFovAngle));
// Create render target.
Sizei recommenedTex0Size = ovr_GetFovTextureSize(session, ovrEye_Left newFovLeft,
pixelsPerDisplayPixel);
Sizei recommenedTex1Size = ovr_GetFovTextureSize(session, ovrEye_Right, newFovRight,
pixelsPerDisplayPixel);
...
// Initialize rendering info.
ovrFovPort eyeFov[2]; eyeFov[0] = newFovLeft; eyeFov[1] = newFovRight;
...
// Determine projection matrices.
ovrMatrix4f projMatrixLeft = ovrMatrix4f_Projection(newFovLeft, znear, zfar, 0); ovrMatrix4f projMatrixRight = ovrMatrix4f_Projection(newFovRight, znear, zfar, 0);
It might be interesting to experiment with non-square fields of view. For example, clamping the up and down ranges significantly (e.g. 70 degrees FOV) while retaining the full horizontal FOV for a ‘Cinemascope’ feel.
OculusWorldDemo allows you to experiment with reducing the FOV below the defaults. Navigate to the settings menu (press the Tab key) and select the “Max FOV” value. Pressing the up and down arrows to change the maximum angle in degrees.
Oculus Rift | Advanced Rendering Configuration | 37
Improving Performance by Rendering in Mono
A significant cost of stereo rendering is rendering two views, one for each eye.
For some applications, the stereoscopic aspect may not be particularly important and a monocular view might be acceptable in return for some performance. In other cases, some users may get eye strain from a stereo view and wish to switch to a monocular one. However, they still wish to wear the HMD as it gives them a high FOV and head-tracking.
OculusWorldDemo allows the user to toggle mono render mode by pressing the F7 key.
To render in mono, your code should have the following changes:
• Set the FOV to the maximum symmetrical FOV based on both eyes.
• Call ovhHmd_GetFovTextureSize with this FOV to determine the recommended render target size.
• Configure both eyes to use the same render target and the same viewport when calling ovr_SubmitFrame.
• Render the scene once to the shared render target.
This merges the FOV of the left and right eyes into a single intermediate render. This render is still distorted twice, once per eye, because the lenses are not exactly in front of the user’s eyes. However, this is still a significant performance increase.
Setting a virtual IPD to zero means that everything will seem gigantic and infinitely far away, and of course the user will lose much of the sense of depth in the scene.
Note: It is important to scale virtual IPD and virtual head motion together so, if the virtual IPD is set to zero, all virtual head motion due to neck movement is also be eliminated. Sadly, this loses much of the depth cues due to parallax. But, if the head motion and IPD do not agree, it can cause significant disorientation and discomfort. Experiment with caution!
Protecting Content
There are some cases where you only want the content to display on the headset. The protected content feature is designed to prevent any mirroring of the compositor.
To use the protected content feature, configure your application to create one or more ovrTextureSwapChain objects with the ovrTextureMisc_ProtectedContent flag specified. Any submission which references a protected swap chain is considered a protected frame; any protected frames are composited and displayed in the HMD, but are not replicated to mirrors or available to the capture buffer API.
If the HMD is not HDCP compliant, the texture swap chain creation API will fail with ovrError_ContentProtectionNotAvailable. If the textures can be created (HDCP-compliant HMD), but the link is broken later, the next ovrSubmitFrame call that references protected texture swap chains will fail with the ovrError_ContentProtectionNotAvailable error. Configure your application to respond according to your requirements. For example, you might submit that next frame without protected swap chains, but at a lower quality that doesn’t require protection. Or, you might stop playback and display an error or warning to the user.
Note: Because the mirror window is in the control of the application, if your application does not use the mirror texture, it is up to your application to render something to the preview window. Make sure your application does not display protected content. Since the surfaces are not known to be protected by the OS, they will be displayed normally inside the application that created them.
38 | Advanced Rendering Configuration | Oculus Rift
To enable protected content, specify the ovrTextureMisc_ProtectedContent flag similarly to the following: ovrTextureSwapChainDesc desc = {}; desc.Type = ovrTexture_2D; desc.ArraySize = 1; desc.Format = OVR_FORMAT_R8G8B8A8_UNORM_SRGB; desc.Width = sizeW; desc.Height = sizeH; desc.MipLevels = 1; desc.SampleCount = 1; desc.MiscFlags = ovrTextureMisc_DX_Typeless | ovrTextureMisc_ProtectedContent; desc.BindFlags = ovrTextureBind_DX_RenderTarget; desc.StaticImage = ovrFalse; ovrResult result = ovr_CreateTextureSwapChainDX(session, Device, &desc, &TextureChain);
Oculus Rift | VR Focus Management | 39
VR Focus Management
When you submit your application to Oculus, you provide the application and metadata necessary to list it in the Oculus Store and launch it from Oculus Home.
Once launched from Oculus Home, you need to write a loop that polls for session status. ovr_GetSessionStatus returns a struct with the following booleans:
•
ShouldQuit—True if the application should initiate shutdown.
•
HmdPresent—True if an HMD is present.
•
DisplayLost—True if the HMD was unplugged or the display driver was manually disabled or encountered a TDR.
•
HmdMounted—True if the HMD is on the user's head.
•
IsVisible—True if the game or experience has VR focus and is visible in the HMD.
•
ShouldRecenter—True if the application should call ovr_RecenterTrackingOrigin. This is triggered when the user initiates recentering through the Universal Menu.
Managing When a User Quits
If
ShouldQuit is true, save the application state and shut down or shut down without saving the application state. The user will automatically return to Oculus Home.
Depending on the type of application, you can prompt the user to start where he or she left off the next time it is opened (e.g., a multi-level game) or you can just start from the beginning of the experience (e.g., a passive video). If this is a multiplayer game, you might want to quit locally without ending the game.
Managing When a User Requests Recentering
If
ShouldRecenter is true, the application should call ovr_RecenterTrackingOrigin or ovr_SpecifyTrackingOrigin and be prepared for future tracking positions to be based on a different origin.
Some applications may have reason to ignore the request or to implement it via an internal mechanism other than via ovr_RecenterTrackingOrigin. In such cases the application can call ovr_ClearShouldRecenterFlag to cause the recenter request to be cleared.
Managing an Unplugged Headset
If
DisplayLost is true:
1. Pause the application, including audio.
2. Display a prompt on the monitor that says that the headset was unplugged.
3. Destroy any TextureSwapChains or mirror textures.
4. Call ovrDestroy.
5. Poll ovrSessionStatus::HmdPresent until true.
6. Call ovrCreate to recreate the session.
7. Recreate any TextureSwapChains or mirror textures.
8. Resume the application.
If ovrDetect doesn’t isn’t returned as true after a specified amount of time, act as though ShouldQuit returned true. If the user takes no action after a specified amount of time, choose a default action (save the session or close without saving) and close the application.
40 | VR Focus Management | Oculus Rift
Note: For multiplayer games, you might want to follow the same process without pausing the game.
Managing an Unavailable Headset
When a user removes the headset or if your application does have VR focus,
HmdMounted or IsVisible returns false. Pause the application until they return true.
When your application loses VR focus, it automatically stops receiving input. If your application does not use the Oculus input API, it will need to ignore any received input.
Note: For multiplayer games, you might want the game to continue without pausing.
Managing Loss of Windows Focus
When your application loses Windows focus, the Oculus Remote, Xbox controller, and Touch controllers will continue to work normally. However, the application will lose control of the mouse and keyboard.
If your application loses Windows focus and maintains VR focus (
IsVisible), continue to process input and allow the application to run normally. If the keyboard or mouse is needed to continue, prompt the user to remove the headset and use Alt-Tab to regain Windows focus.
Code Sample bool shouldQuit = false; void RunApplication()
{
ovrResult result = ovr_Initialize();
if (OVR_SUCCESS(result))
{
ovrSession session;
ovrGraphicsLuid luid;
result = ovr_Create(&session, &luid);
if (OVR_SUCCESS(result))
{
ovrSessionStatus ss;
<create graphics device with luid>
<create render target via ovr_CreateTextureSwapChain>
while (!shouldQuit)
{
<get next frame pose, e.g. via ovr_GetEyePoses>
<render frame>
result = ovr_SubmitFrame(...);
if (result == ovrSuccess_NotVisible)
{
<turn off audio output>
do { // Wait until we regain visibility or should quit
<sleep>
result = ovr_GetSessionStatus(session, &ss);
if (ss.ShouldQuit)
shouldQuit = true;
} while (OVR_SUCCESS(result) && !ss.IsVisible && !shouldQuit);
<possibly re-enable audio>
}
else if (result == ovrError_DisplayLost)
{
// We can either immediately quit or do the following:
<destroy render target and graphics device>
ovr_Destroy(session);
Oculus Rift | VR Focus Management | 41
do { // Spin while trying to recreate session.
result = ovr_Create(&session, &luid);
} while (OVR_FAILURE(result) && !shouldQuit);
if (OVR_SUCCESS(result))
{
<recreate graphics device with luid>
<recreate render target via ovr_CreateTextureSwapChain>
}
}
else if (OVR_FAILURE(result))
{
shouldQuit = true;
}
ovr_GetSessionStatus(session, &ss);
if (ss.ShouldQuit)
shouldQuit = true;
if (ss.ShouldRecenter)
{
ovr_RecenterTrackingOrigin(session); // or ovr_ClearShouldRecenterFlag(session)
to ignore the request.
<do anything else needed to handle this>
}
}
<destroy render target via ovr_DestroyTextureSwapChain>
<destroy graphics device>
ovr_Destroy(session);
}
ovr_Shutdown();
}
}
42 | Oculus Guardian System | Oculus Rift
Oculus Guardian System
The Oculus Guardian System is designed to display in-application wall and floor markers when users get near boundaries they defined. When the user gets too close to the edge of a boundary, translucent boundary markers are displayed in a layer that is superimposed over the game or experience.
The following image shows the Guardian System activated. Note the floor boundaries (lines) and the wall boundaries (crosses):
Figure 11: Oculus Boundary System
Setting Up the Guardian System on Your Rift
After the user sets up the boundaries, they will show up in a layer over any application whenever the user gets too close.
Note: Users can set the boundaries to Walls and Floor, set them to Floor Only, or disable them entirely.
To set up the boundaries:
1. Open the Oculus App.
2. Select Settings -> Devices -> Run Full Setup.
3. Select Rift and Touch.
Oculus Rift | Oculus Guardian System | 43
4. Follow the on-screen instructions to confirm sensor tracking.
5. Continue until the Mark Your Boundaries page appears.
6. Follow the on-screen instructions, using the INDEX trigger button to draw the outer bounds of your play area. Currently, there is no minimum width and depth.
When you are finished, click Next. Your boundaries are saved.
7. If you need to disable the Guardian System, toggle Guardian System Enabled/Disabled on the Universal
Menu.
Game Configuration
During initialization, your application can make an API request to get the outer boundary and play area. The outer boundary is the space that the user defined during configuration. The play area is a rectangular space within the outer boundary. With this information, your application can set up a virtual world with "barriers" that align with the real world. For example, you can adjust the size of a cockpit based on the user-defined play area.
The following functions return information about the outer boundary and play area:
Function ovr_GetBoundaryDimensions ovr_GetBoundaryGeometry
Description
Returns the width, height, and depth of the play area or outer boundary in meters.
Returns the points that define the play area or outer boundary. For the play area, it returns the four points that define the rectangular area. For the outer boundary, it returns all of the points that define the outer boundary.
During runtime, your application can request whether the boundaries are visible using ovr_GetBoundaryVisible. When visible, you can choose how the application will respond. For example, you might choose to pause the application, slow the application, or simply display a message.
The boundary status information is returned in a struct which contains the following:
Member
IsTriggering
ClosestDistance
ClosestPoint
ClosestPointNormal
Type ovrBool float ovrVector3f ovrVector3f
Description
Returns whether the boundaries are currently visible.
Distance to the closest play area or outer boundary surface.
Closest point on the boundary surface.
Unit surface normal of the closes boundary surface.
Additionally, you can set the bounds to be visible to orient the user or explain how you will use the space by setting ovr_RequestBoundaryVisible() to ovrTrue. When you are finished, simply pass ovrFalse.
Note: You can't force the boundaries off if they were triggered by user.
The default boundary color is cyan. To change the color, use
ovr_SetBoundaryLookAndFeel().
44 | Oculus Guardian System | Oculus Rift
Code Sample
To help you get started, we provide a code sample at
Samples/GuardianSystemDemo that shows usage of the following APIs:
• ovr_TestBoundary
• ovr_TestBoundaryPoint
• ovr_SetBoundaryLookAndFeel
• ovr_RequestBoundaryVisible
• ovr_ResetBoundaryLookAndFee
Boxes collide with the boundary data using the test API, the boundary visibility and color changes every second, and the simulation slows (and then stops) when the HMD or Touch controllers get too close to the boundary.
Oculus Rift | Rift Audio | 45
Rift Audio
When setting up audio for the Rift, you need to determine whether the Rift headphones are active and pause the audio when your app doesn’t have focus.
The user can enable the Rift headphones and microphone in the Oculus App or use the default Windows audio devices. Configuration is made the Oculus app in Settings -> Devices -> Headset. The following screenshot shows the Rift headphones disabled and the microphone enabled:
Figure 12: Audio Configuration
The headphone setting is handled automatically by the function ovr_GetAudioDeviceOutGuid (located in OVR_CAPI_Audio.h), which returns the GUID for the device to target when playing audio. Similarly, use ovr_GetAudioDeviceInGuid to identify the microphone device used for input.
46 | Rift Audio | Oculus Rift
FMOD D for Native Rift Development
If you detect that the Rift headphones are enabled, use code similar to the following for FMOD:
ERRCHECK(FMOD::System_Create(&sys));
GUIDguid; ovr_GetAudioDeviceOutGuid(&guid); intdriverCount=0; sys->getNumDrivers(&driverCount); intdriver=0; while(driver<driverCount)
{
charname[256]={0};
FMOD_GUIDfmodGuid={0};
sys->getDriverInfo(driver,name,256,&fmodGuid,nullptr,nullptr,nullptr);
if(guid.Data1==fmodGuid.Data1&&
guid.Data2==fmodGuid.Data2&&
guid.Data3==fmodGuid.Data3&&
memcmp(guid.Data4,fmodGuid.Data4,8)==0)
{
break;
}
++driver;
} if(driver<driverCount)
{
sys->setDriver(driver);
} else
{
// error rift not connected
}
For instructions on using FMOD with Unreal Engine, see “Unreal and FMOD” below.
Wwise for Native Rift Development
If you detect that the Rift headphones are enabled, use code similar to the following for Wwise:
AkInitSettings initSettings;
AkPlatformInitSettings platformInitSettings;
AK::SoundEngine::GetDefaultInitSettings( initSettings );
AK::SoundEngine::GetDefaultPlatformInitSettings( platformInitSettings );
// Configure initSettings and platformInitSettings...
WCHAR outStr[128]; if (OVR_SUCCESS(ovr_GetAudioDeviceOutGuidStr(outStr)))
{
initSettings.eMainOutputType = AkAudioAPI::AkAPI_Wasapi;
platformInitSettings.idAudioDevice = AK::GetDeviceIDFromName(outStr);
}
For instructions on using Wwise with Unity 5, see “Unity 5 and Wwise” below. For instructions on using Wwise with Unreal Engine, see “Unreal and FMOD” below.
Oculus Rift | Rift Audio | 47
Unity 5
Audio input and output automatically use the Rift microphone and headphones unless configured to use the
Windows default audio device by the user in the Oculus app. Events
OVRManager.AudioOutChanged and
AudioInChanged occur when audio devices change, making audio playback impossible without a restart.
Unity 5 and Wwise
To configure Wwise to use to configured audio device in Unity 5, pass the user-configured audio device name/
GUID (set in the Oculus app) into the function
AkSoundEngine.GetDeviceIDFromName(), located in
AkInitializer.cs.
To get the audio device GUID from libOVR, you must include the Oculus Utilities unitypackage, which exposes that string through the class OVRManager.
The following function should be called before
AkSoundEngine.Init(...): void SetRiftAudioDevice(AkPlatformInitSettings settings)
{ string audioDevice = OVRManager.audioOutId; uint audioOutId = AkSoundEngine.GetDeviceIDFromName (audioDevice); settings.idAudioDevice = audioOutId;
}
Pass
AkPlatformInitSettings into the function above and use it to initialize the Ak sound engine.
Note:
OVRManager.audioOutId will be deprecated in the future. This minor change should not impact the integration.
VR Audio Output in Oculus Store > Settings > Devices > Rift Headset may be used to configure which input mic to use within the Ak sound engine. The GUID for this is exposed through
OVRManager.audioInId.
Unity 4
Audio input and output automatically use the Rift microphone and headphones, unless configured to use the
Windows default audio device by the user in the Oculus app.
The Rift’s microphone is not used by default when calling
Microphone.Start(null,..). Find the entry in
Microphone.devices that contains the word Rift and use it.
Unreal
When Unreal PC applications are launched, if the OculusRift plugin is enabled and the Oculus VR Runtime
Service is installed, then the application will automatically override the default Windows graphics and audio devices and target the Rift. The Oculus VR Runtime Service is installed with the Oculus App.
Unless your application is intended to run in VR, do not enable the OculusRift plugin. Otherwise, it is possible that audio and/or video will be incorrectly targeted to the Oculus Rift when the application is run.
Alternatively, users can disable loading all HMD plugins by specifying "-nohmd" on the command line.
Unreal and Wwise
To use Wwise with any Unreal version, use the Wwise Unreal Plug-in provided by Audiokinetic here:
https:// www.audiokinetic.com/library/2016.1.0_5775/?source=UE4&id=index.html
.
Note: The link above links to build Wwise 2016.1 build 5775 for use with Unity 4.12. Be sure to select the appropriate build for your Unity version in the upper-left.
48 | Rift Audio | Oculus Rift
Unreal and FMOD
For an illustration of how to target the Oculus Rift headphones using FMOD in UE4, see the FMOD section above.
Oculus Audio Spatialization
The Oculus Audio SDK, available from our Downloads page, provides free, easy-to-use spatializer plugins for engines and middleware. Our spatialization features support both Rift and Gear VR development.
For a detailed discussion of audio spatialization and virtual reality audio, we recommend beginning with our
Introduction to Virtual Reality Audio and Oculus Audio SDK guides.
For additional information on using Oculus spatializer plugins with Unity and Unreal, see Unity Audio and
Unreal VR Audio.
VR Sound Level Best Practices
Audio is an important part of the virtual reality experience, and it is important that all VR apps provide a comfortable listening level for users. Developers should target a reasonable sound level that is consistent between different experiences. To achieve this, Oculus recommends the following best practices.
First, target -18 LUFS during final mix, using tools such as Avid's Pro Limited Plugin, Nugen's VisLM,
Klangfreund LUFS Meter, Audacity VuMeter, or similar loudness measurement tools.
Second, measure your experience against the audio levels in published Oculus experiences, especially the ambient audio in Home and the Dreamdeck experience. Set overall system volume so that Dreamdeck and
Home sound comfortable, and then adjust your experience's mix to this volume.
Finally, mix your application using the Rift headphones. This ensures that the sounds you're creating and mixing have a frequency content appropriate for the headphones most Oculus users will use.
By adhering to these guidelines, we can guarantee that our Oculus VR users will have a pleasant audio experience.
Oculus Rift | Oculus Touch Controllers | 49
Oculus Touch Controllers
This section describes Oculus Touch best practices gathered from developing and reviewing large numbers of games and experiences. These are not requirements and we expect them to evolve over time.
Note: To view application requirements, go to
https://developer.oculus.com/distribute/latest/concepts/ publish-rift-app-submission/
Input, Hands, and Controller Basics
When developing applications that use Oculus Touch, consider the following:
• If your application supports multiple input methods (such as Oculus Touch, Xbox 360 Controller, and Oculus
Remote), use ovrControllerType_Active to determine which one is in use. You can also use this to determine if one or both Touch controllers are in use. Some applications render the Touch controllers differently (e.g. as hands vs. controllers) depending on their in-use state. For more information, see this section and
https://developer.oculus.com/doc/1.14-libovr/_o_v_r___c_a_p_i_8h.html
.
• Unless you have an uncommon use case, use the Avatar SDK to represent high quality hands and/or controllers in your app.
• Map the grip button to grab actions. Although there are some exceptions to the rule (especially involving throwing), the new user experience and most applications condition users to use it this way. If you break expectations, make sure to educate your users.
• Prefer hands over controller models, especially for any application that involves social interaction or picking things up.
• In-application hands and controllers should line up with the user’s real-world counterparts in position and orientation as closely as possible. We call this “registration” and recommend the Avatar SDK as an example.
When testing your application, one technique is to hold your hands in front of your face and raise your headset slightly so you can compare your real-world hands and your virtual hands.
• Fully animate all hands and controllers in the game, to reflect what the user is doing. Users will expect to be able to grab, point, and thumbs-up with their hands. They will expect controller joysticks, buttons, and triggers to animate on use. We have demos with full code and blueprints for doing this in UE4 and Unity.
• To support left- and right-handed players, we recommend designing objects for use with either hand. For example, if the player is going to use a double rotating-wheel can opener to open a can in VR, make sure the cutting wheel handle faces the other hand.
• When controllers are mirrored, make controls identical. When controllers have different functions, make sure to establish a dominant and non-dominant hand (instead of favoring the right hand). For example, in the Toy
Box demo, players usually use their non-dominant to hold the slingshot and their dominant hand to pull the ammunition holder.
• Unless important to the experience, do not render additional body parts besides the hands in single player games. Although inverse kinematics (i.e., extrapolating body part positions based on how the body can and cannot move) are a tempting solution to represent more the player’s body, they become less accurate as you try to simulate more of the body. The mismatch can end up being distracting and ironically less immersive.
For multiplayer games, you can render the hands of the first person and the up to the full bodies of the other players.
Tracking
The following are basic tracking tips:
• For front-facing apps, keep the action in front of the user. Don’t encourage users to do things that will break the line of sight between the controllers and sensors, which will lead to poor tracking.
50 | Oculus Touch Controllers | Oculus Rift
• Don’t require the user to interact with VR elements near the floor or far above their heads. Remove those objects, or allow a distance grab. If you decide to allow interactions at a distance, make sure to indicate that the object is available through a highlight, glow, shake, haptics, or another mechanism.
• Avoid interactions that encourage users to get too close or too far from one or more position trackers.
Large Play Areas
For applications that use a large amount of the tracking space, either horizontally or vertically, you’ll want to put a little more effort into making the best use of the available space:
• Our user-facing description of “Standing” apps is “may require a step in any direction.” Even if your application uses a lot of space, if it is going to be submitted as a standing app, it needs to meet that description. Additionally, Standing apps must be fully usable in a 2m x 1.5 tracking area.
• If your application requires lots of space, use the user’s Guardian boundaries to make sure you place game
objects within reachable areas. For more information, see
• Applications can modify recenter behavior in a way that makes sense for the app. For example, an app might clamp the recenter origin inside of the play area by some amount of padding, so that all gameplay elements are reachable. Or it might choose to ignore the yaw component, to maintain an axis-aligned play area. If an app substantially violates user expectations for recenter, it should inform the user about what’s happening. For more information, see
on page 39.
• Applications can also use the above recenter functionality to set an optimal center position (overriding the user’s home position) on launch. You can use the default recenter behavior on subsequent user-initiated recenters, or continue to use modified recenter logic.
• If the user’s play space is undefined, create your own play space around the user’s recenter/home point.
This assumed play space should not exceed 2m x 1.5m (the recommended play space size). If your application can use a smaller space, default to that.
• If your app requires high or low tracking, keep the shape of the sensor frustums in mind. For example, a front-facing app that involves shooting basketballs will want to keep the user towards the rear of the play area, where the tracking frustums are taller. For more information on sensor field of view and tracking
Initialization and Sensor Enumeration
• Some users might not have the full 2m x 1.5m recommended play space, but will play your game despite warnings. If your application requires this full amount of space, choose the best fallback. Possible solutions include:
• It is generally better to overflow away from the sensors, not towards, since tracking is better further back
(and closer might put you past the sensor position).
• Users can attempt to use recenter to reach playable areas they would otherwise be unable to reach.
Controller Data
The Oculus SDK provides APIs that return the position and state for each Oculus Touch controller.
This data is exposed through two locations:
• ovrTrackingState::HandPoses[2]—returns the pose and tracking state for each Oculus Touch controller.
• ovrInputState—structure returned by ovr_GetInputState that contains the Oculus Touch button, joystick, trigger, and capacitive touch sensor state.
The controller hand pose data is separated from the input state because it comes from a different system and is reported at separate points in time. Controller poses are returned by the constellation tracking system and are predicted simultaneously with the headset, based on the absolute time passed into GetTrackingState. Having both hand and headset data reported together provides a consistent snapshot of the system state.
Oculus Rift | Oculus Touch Controllers | 51
Hand Tracking
The constellation sensor used to track the head position of the Oculus Rift also tracks the hand poses of the
Oculus Touch controllers.
For installations that have the Oculus Rift and Oculus Touch controllers, there will be at least two constellation sensors to improve tracking accuracy and help with occlusion issues.
The SDK uses the same ovrPoseStatef struct as the headset, which includes six degrees of freedom (6DoF) and tracking data (orientation, position, and their first and second derivatives).
Here’s an example of how to get tracking input: ovrTrackingState trackState = ovr_GetTrackingState(session, displayMidpointSeconds, ovrTrue); ovrPosef handPoses[2]; ovrInputState inputState;
In this code sample, we call ovr_GetTrackingState to get predicted poses. Hand controller poses are reported in the same coordinate frame as the headset and can be used for rendering hands or objects in the 3D world.
An example of this is provided in the Oculus World Demo.
Button State
The input button state is reported based on the HID interrupts arriving to the computer and can be polled by calling ovr_GetInputState.
The following example shows how input can be used in addition to hand poses: double displayMidpointSeconds = ovr_GetPredictedDisplayTime(session, frameIndex); ovrTrackingState trackState = ovr_GetTrackingState(session, displayMidpointSeconds, ovrTrue); ovrPosef handPoses[2]; ovrInputState inputState;
// Grab hand poses useful for rendering hand or controller representation handPoses[ovrHand_Left] = trackState.HandPoses[ovrHand_Left].ThePose; handPoses[ovrHand_Right] = trackState.HandPoses[ovrHand_Right].ThePose; if (OVR_SUCCESS(ovr_GetInputState(session, ovrControllerType_Touch, &inputState)))
{
if (inputState.Buttons & ovrButton_A)
{
// Handle A button being pressed
}
if (inputState.HandTrigger[ovrHand_Left] > 0.5f)
{
// Handle hand grip...
}
}
The ovrInputState struct includes the following fields:
Field
TimeInSeconds
Type double
ControllerType unsigned int
Description
System time when the controller state was last updated.
Described by ovrControllerType.
Indicates which controller types
52 | Oculus Touch Controllers | Oculus Rift
Field Type
Buttons
Touches
IndexTrigger[2]
HandTrigger[2]
Thumbstick[2]
IndexTriggerNoDeadzone[2] float unsigned int unsigned int float float ovrVector2f
Description are present; you can check the ovrControllerType_LTouch bit, for example, to verify that the left touch controller is connected.
Options include:
• ovrControllerType_None
(0x0000)
• ovrControllerType_LTouch
(0x0001)
• ovrControllerType_RTouch
(0x0002)
• ovrControllerType_Touch
(0x0003)
• ovrControllerType_Remote
(0x0004)
• ovrControllerType_XBox
(0x0010)
Button state described by ovrButtons. A corresponding bit is set if the button is pressed.
Touch values for buttons and sensors as indexed by ovrTouch.
A corresponding bit is set if users finger is touching the button or is in a gesture state detectable by the controller.
Left and right finger trigger values
(ovrHand_Left and ovrHand_Right), in the range 0.0 to 1.0f. A value of
1.0 means that the trigger is fully pressed.
Left and right grip button values
(ovrHand_Left and ovrHand_Right), in the range 0.0 to 1.0f. Hand trigger is often used to grab items.
A value of 1.0 means that the trigger is fully pressed.
Horizontal and vertical thumbstick axis values (ovrHand_Left and ovrHand_Right), in the range -1.0f
to 1.0f. The API automatically applies the dead zone, so developers don’t need to handle it explicitly.
Left and right finger trigger values
(ovrHand_Left and ovrHand_Right), in the range 0.0 to 1.0f, without a deadzone. A value of 1.0 means that the trigger is fully pressed.
Oculus Rift | Oculus Touch Controllers | 53
Field
HandTriggerNoDeadzone[2]
ThumbstickNoDeadzone[2]
IndexTriggerRaw[2]
HandTriggerRaw[2]
ThumbstickRaw[2]
Type float ovrVector2f float float ovrVector2f
Description
Left and right grip button values
(ovrHand_Left and ovrHand_Right), in the range 0.0 to 1.0f, without a deadzone. The grip button, formerly known as the hand trigger, is often used to grab items. A value of 1.0 means that the button is fully pressed.
Horizontal and vertical thumbstick axis values (ovrHand_Left and ovrHand_Right), in the range -1.0f
to 1.0f, without a deadzone.
Raw left and right grip button values (ovrHand_Left and ovrHand_Right), in the range 0.0 to
1.0f, without a deadzone or filter. A value of 1.0 means that the trigger is fully pressed.
Left and right grip button values
(ovrHand_Left and ovrHand_Right), in the range 0.0 to 1.0f, without a deadzone or filter. The grip button, formerly known as the hand trigger, is often used to grab items. A value of 1.0 means that the button is fully pressed.
Horizontal and vertical thumbstick axis values (ovrHand_Left and ovrHand_Right), in the range -1.0f
to 1.0f, without a deadzone or filter.
The ovrInputState structure includes the current state of buttons, thumb sticks, triggers and touch sensors on the controller. You can check whether a button is pressed by checking against one of the button constants, as was done for ovrButton_A in the above example. The following is a list of binary buttons available on touch controllers:
Button Constant ovrButton_A ovrButton_B ovrButton_RThumb ovrButton_X ovrButton_Y ovrButton_LThumb ovrButton_Enter
Description
A button on the right Touch controller.
B button on the right Touch controller.
Thumb stick button on the right Touch controller.
X button on the left Touch controller.
Y button on the left Touch controller.
Thumb stick button on the left Touch controller.
Enter button on the left Touch controller. This is equivalent to the Start button on the Xbox controller.
54 | Oculus Touch Controllers | Oculus Rift
Button Touch State
In addition to buttons, Touch controllers can detect whether user fingers are touching some buttons or are in certain positions.
These states are reported as bits in the Touches field, and can be checked through one of the following constants: ovrTouch_A ovrTouch_B ovrTouch_RThumb ovrTouch_RThumbRest ovrTouch_RIndexTrigger ovrTouch_X ovrTouch_Y ovrTouch_LThumb ovrTouch_LThumbRest ovrTouch_LIndexTrigger ovrTouch_RIndexPointing ovrTouch_RThumbUp ovrTouch_LIndexPointing ovrTouch_LThumbUp
User in touching A button on the right controller.
User in touching B button on the right controller.
User has a finger on the thumb stick of the right controller.
User has a finger on the textured thumb rest of the right controller.
User in touching the index finger trigger on the right controller.
User in touching X button on the left controller.
User in touching Y button on the left controller.
User has a finger on the thumb stick of the left controller.
User has a finger on the textured thumb rest of the left controller.
User in touching the index finger trigger on the left controller.
Users right index finger is pointing forward past the trigger.
Users right thumb is up and away from buttons on the controller, a gesture that can be interpreted as right thumbs up.
Users left index finger is pointing forward past the trigger.
Users left thumb is up and away from buttons on the controller, a gesture that can be interpreted as left thumbs up.
Haptic Feedback
In addition to reporting input state, Oculus touch controllers can provide haptic feedback through vibration.
The SDK supports two types of haptics:
•
Non-Buffered Haptics – With this approach, haptics are controlled by simply turning vibrations on and off, while specifying a frequency (160Hz or 320Hz) and an amplitude (0 to 255). Non-buffered haptics are designed for simple effects that don't have tight latency requirements, since the controller requires 33ms to respond to the API call that modifies the haptics settings.
Oculus Rift | Oculus Touch Controllers | 55
•
Buffered Haptics – This approach enables a much wider variety of effects, such as patterning vibrational amplitudes around sine wave or tangent functions, panning the vibrations across controllers, generating a variety of low-frequency carrier waves, and more. With buffered haptics, your application sends a buffer of bytes to one or both controllers, where the bytes are interpreted as a series of amplitude values, from 0 (no vibrational amplitude) to 255 (maximum vibrational amplitude). The controller then “plays” the buffered amplitude values at 320Hz (or one byte every 3.125ms). Thus, you can control the vibrational amplitudes at a much finer degree of granularity when compared to non-buffered haptics (although it requires about 10ms from the time of the API call until the buffer is available to be played within the controller).
Note: The buffered and non-buffered functions should not be used together, as they will result in unpredictable haptic feedback.
Using Non-Buffered Haptics
Vibration can be enabled by calling ovr_SetControllerVibration: ovr_SetControllerVibration( Hmd, ovrControllerType_LTouch, freq, trigger);
Vibration is enabled by specifying the frequency. Specifying 0.0f will vibrate at 160Hz. Specifying 1.0f will vibrate at 320Hz.
Buffered Haptics Overview
Buffered haptics enable you to create many cool effects beyond what is possible with non-buffered haptics. For example, the buffered haptics sample app that is provided with the PC-SDK (and which is described later in this section) produces the following effects:
• Smooth sine wave vibration with a "buzz down" effect at the end of each wave cycle
• Vibrational panning across the left and right controllers, again with a "buzz down" effect at the end of the panning cycle
• Ultra low-frequency buzz, essentially a series of ticks at 64Hz
• A "messed up" low frequency vibration based on a chaotic formula that utilizes a trigonometric tangent wave function
A buffer consists of a series of bytes with values from 0 to 255, where 0 represents no amplitude (i.e no vibration), and 255 represents the maximum amplitude (or intensity) of vibration that is allowed by the SDK.
After your code fills in the values within a buffer, you send the buffer to one or both Touch controllers via ovr_SubmitControllerVibration. Each byte in the buffer is then "played" in sequence at a rate of 320Hz.
The maximum buffer size (i.e. the maximum number of bytes that can be sent to a controller at one time, and also the maximum size of the controller's internal buffer) is 256 bytes. The length of time that it takes to "play" a single 256 byte buffer is 0.8 seconds (256 bytes played at a rate of 320Hz). So, you have full control over the amplitude of the vibrational effects down to a resolution of 3.125ms (which equates to 320Hz). However, the frequency can only be 320Hz or some integral quotient of 320Hz, such as 320/2=160Hz, 320/3=106.7Hz,
320/4=80Hz, 320/5=64Hz, etc. You can achieve these lower frequencies by sending bytes that are zero filled, interspersed with bytes that have amplitude values that are greater than zero. Here are some examples:
• 320Hz, full amplitude - [255, 255, 255, 255, ...]
• 160Hz, full amplitude - [255, 0, 255, 0, 255, 0, 255, 0, ...]
• 320Hz, half amplitude - [127, 127, 127, ..., 127, ...]
• 160Hz, half amplitude - [127, 0, 127, 0, 127, 0, …, 127, 0, ...]
• Single sharp tick (320Hz) - [0, 0, 255, 255, 255, 0, 0] [delay x ms] [0, 0, 255, 255, 255, 0, 0]
• Single blunt tick (160Hz) - [0, 255, 0, 255, 0, 255, 0] [delay x ms] [0, 255, 0, 255, 0, 255, 0]
In general, use the 320Hz resonant mode for lighter, sharper actions and the 160Hz mode for heavier, blunter actions.
56 | Oculus Touch Controllers | Oculus Rift
In the above "sharp tick" and "blunt tick" examples, you can try varying the pulse count anywhere from 1 to
50 to obtain different lengths of effect. You can also try varying the amplitude to obtain different vibrational intensities. In general, you can vary both the amplitude and the frequencies in a more random or chaotic way.
You can also vary the vibrational effects based on input streams, such as controller movement or position, or any other conditions or events within the VR experience.
In addition to the types of effects described above, there are many other possibilities, including:
• Hybrid Ticks and Modulation
You can combine a repeating tick with an amplitude modulated segment.
• Mixing Multiple Input Streams
You can pre-mix multiple input streams prior to passing to the haptics buffer API.
Note: Because of the relatively slow resonant decay of the haptics hardware within the controller, there is going to be a non-linear result based upon previous samples in a buffer. For example, if you want a sample to resonate at 50% amplitude but the previous sample was at 100% amplitude, you might need to drive the amplitude in a different way than if the previous sample was at 0% amplitude. An overdrive algorithm may improve the response. Positive (attack) and negative (decay) likely require different constants. A basic algorithm could simply factor in the sample-to-sample delta. A more advanced algorithm may need to use a weighted average of last several samples.
Note: Once a buffer is sent to a controller (via ovr_SubmitControllerVibration), that entire buffer will be "played". There is no way to halt the playing of a buffer after it has been sent to a controller.
So, you should take care to only buffer up content that you know you wish to play within your VR application.
It is important to keep your sample pipeline at around the right size. Assuming a haptic frequency of 320
Hz and an application frame rate of 90 Hz, we recommend targeting a buffer size of around 10 samples per frame. This allows you to play 3-4 haptics bytes per frame, while preserving a buffer zone to account for any asynchronous interruptions. The more bytes you queue, the safer you are from interruptions, but you add additional latency before newly queued vibrations will be played.
Take care to not overflow or underflow the 256 byte internal buffer within the controller. If you call ovr_SubmitControllerVibration with a larger buffer than the number of bytes that are currently available within the (circular) internal buffer, the entire submitted buffer is discarded. On the other hand, if you submit bytes at a rate that fails to keep up with the consumption of buffered bytes within the controller, there will be gaps in the vibrational playback.
Using Buffered Haptics
To check the status of the buffer, call ovr_GetControllerVibrationState: ovr_GetControllerVibrationState(ovrSession session, ovrControllerType controllerType,
ovrHapticsPlaybackState* outState);
To submit to the buffer, call ovr_SubmitControllerVibration: ovr_SubmitControllerVibration(ovrSession session, ovrControllerType controllerType, const
ovrHapticsBuffer* buffer);
The following code sample shows how to produce some interesting and cool effects. This code extends the basic sample application contained in the Oculus PC-SDK distribution, in the following Visual Studio project
<install_folder>\Samples\OculusRoomTiny_Advance\ORT (Buffered Haptics). You can copy/ paste this code into the project, overwriting main.cpp, if desired.
/// A sample to show vibration generation, using buffered input.
/// Press A to generate a sine wave vibration in the right controller.
Oculus Rift | Oculus Touch Controllers | 57
/// Press B to generate a 320 Hz vibration that pans from the right controller to the left
controller.
/// Press X to generate a low-frequency buzz at 64 hz (320 Hz / 5) in the left controller.
/// Press Y to generate a "messed up" low frequency vibration in the left controller, based on a
tangent function.
/// Hold any of the buttons down in order to repeat the pattern continuously.
/// Note: In order to keep the sample minimal, the Touch controller is not graphically displayed
within the VR scene.
#include "../Common/Win32_DirectXAppUtil.h" // DirectX
#include "../Common/Win32_BasicVR.h" // Basic VR struct BufferedHaptics : BasicVR
{
BufferedHaptics(HINSTANCE hinst) : BasicVR(hinst, L"BufferedHaptics") {}
void MainLoop()
{
Layer[0] = new VRLayer(Session);
// We create haptic buffers that will be associated with the A, B, X, and Y buttons.
int bufferSize = 256;
unsigned char * dataBufferA = (unsigned char *)malloc(bufferSize);
unsigned char * dataBufferBRight = (unsigned char *)malloc(bufferSize);
unsigned char * dataBufferBLeft = (unsigned char *)malloc(bufferSize);
unsigned char * dataBufferX = (unsigned char *)malloc(bufferSize);
unsigned char * dataBufferY = (unsigned char *)malloc(bufferSize);
// Verify that the buffer format is what we expect.
ovrTouchHapticsDesc desc = ovr_GetTouchHapticsDesc(Session, ovrControllerType_LTouch);
if (desc.SampleSizeInBytes != 1) FATALERROR("Our assumption of 1 byte per
element, is no longer valid");
if (desc.SubmitMaxSamples < bufferSize) FATALERROR("Can't handle this many samples");
desc = ovr_GetTouchHapticsDesc(Session, ovrControllerType_RTouch);
if (desc.SampleSizeInBytes != 1) FATALERROR("Our assumption of 1 byte per
element, is no longer valid");
if (desc.SubmitMaxSamples < bufferSize) FATALERROR("Can't handle this many samples");
// Fill dataBufferA with a sine wave amplitude pattern that will smoothly
// rise and fall over a cycle period of 0.8 seconds. The 0.8 value is
// equal to 256/320, where 256 is the number of bytes that we will use
// to define a single sine wave cycle, and the bytes are "played"
// on the controller at 320 Hz. An interesting effect is added by lowering
// the effective frequency in latter half of the cycle by setting
// alternate intensities (amplitudes) to zero. The overall effect is a
// smoothly repeating vibrational pattern that "buzzes down" at the end of
// the wave cycle.
for (int i = 0; i < bufferSize; i++)
{
dataBufferA[i] = (unsigned char)(255.0f*(sin(((3.14159265359f*i) /
((float)bufferSize)))));
if ((i > bufferSize / 2) && (i % 2)) dataBufferA[i] = 0;
}
// Fill dataBufferBRight with values that decrement from 255 down to 0.
// Similarly, fill dataBufferBLeft with values that increment from 0 up to 255.
// An interesting effect is added by lowering the effective frequency in
// latter half of dataBufferBLeft by setting intensities (amplitudes) to zero
// for odd numbered bytes that are not divisible by 3. The overall effect
// is a right-to-left vibrational pan, with a distinct "buzzing down" feeling
// at the end of the panning cycle.
for (int i = 0; i < bufferSize; i++)
{
dataBufferBRight[i] = (unsigned char)(255.0f*(255 - i / ((float)bufferSize)));
dataBufferBLeft[i] = (unsigned char)(255.0f*(i / ((float)bufferSize)));
if ((i > bufferSize / 2) && ((i % 2) || (i % 3))) dataBufferBLeft[i] = 0;
}
// Fill dataBufferX with zeros, and set every fifth byte to 255. This creates
// maximum amplitude ticks at 64 Hz. (This is calculated by dividing
// the number of ticks in the buffer, 256/5 = 51.2, and dividing that
// by the time period over which the buffer is played, 256/320 of a second,
// which is 0.8 seconds.)
for (int i = 0; i < bufferSize; i++)
{
dataBufferX[i] = (unsigned char)0;
if (i % 5 == 0) dataBufferX[i] = 255;
}
// Fill dataBufferY with a tangent wave function that varies the intensity
58 | Oculus Touch Controllers | Oculus Rift
// amplitude) between 0 and 255, but set every other byte to 0. This produces a
// "messed up" low frequency vibration.
for (int i = 0; i < bufferSize; i++)
{
dataBufferY[i] = (unsigned char)0;
if (i % 2 == 0) dataBufferY[i] = (unsigned char)
(255.0f*(tan(((3.14159265359f*i) / ((float)bufferSize)))));
}
// Create the SDK structures that contain the buffers, and prepare
// them to be submitted to the controllers.
ovrHapticsBuffer bufferX;
bufferX.SubmitMode = ovrHapticsBufferSubmit_Enqueue;
bufferX.SamplesCount = bufferSize;
bufferX.Samples = (void *)dataBufferX;
ovrHapticsBuffer bufferY;
bufferY.SubmitMode = ovrHapticsBufferSubmit_Enqueue;
bufferY.SamplesCount = bufferSize;
bufferY.Samples = (void *)dataBufferY;
ovrHapticsBuffer bufferA;
bufferA.SubmitMode = ovrHapticsBufferSubmit_Enqueue;
bufferA.SamplesCount = bufferSize;
bufferA.Samples = (void *)dataBufferA;
ovrHapticsBuffer bufferBRight;
ovrHapticsBuffer bufferBLeft;
bufferBRight.SubmitMode = ovrHapticsBufferSubmit_Enqueue;
bufferBRight.SamplesCount = bufferSize;
bufferBRight.Samples = (void *)dataBufferBRight;
bufferBLeft.SubmitMode = ovrHapticsBufferSubmit_Enqueue;
bufferBLeft.SamplesCount = bufferSize;
bufferBLeft.Samples = (void *)dataBufferBLeft;
// Main Loop
while (HandleMessages())
{
Layer[0]->GetEyePoses();
// Submit the haptic buffers to "play" upon pressing A, B, X or Y buttons.
ovrInputState inputState;
ovr_GetInputState(Session, ovrControllerType_Touch, &inputState);
if (inputState.Buttons & ovrTouch_A)
{
// Only submit the buffer if there is enough space available.
ovrHapticsPlaybackState playbackState;
ovrResult result = ovr_GetControllerVibrationState(Session,
ovrControllerType_RTouch, &playbackState);
if (playbackState.RemainingQueueSpace >= bufferSize)
{
ovr_SubmitControllerVibration(Session, ovrControllerType_RTouch,
&bufferA);
}
}
if (inputState.Buttons & ovrTouch_B)
{
// Only submit the buffers if there is enough space available.
ovrHapticsPlaybackState playbackState;
ovrResult result = ovr_GetControllerVibrationState(Session,
ovrControllerType_LTouch, &playbackState);
if (playbackState.RemainingQueueSpace >= bufferSize)
{
ovr_SubmitControllerVibration(Session, ovrControllerType_LTouch,
&bufferBLeft);
}
result = ovr_GetControllerVibrationState(Session,
ovrControllerType_RTouch, &playbackState);
if (playbackState.RemainingQueueSpace >= bufferSize)
{
ovr_SubmitControllerVibration(Session, ovrControllerType_RTouch,
&bufferBRight);
}
}
if (inputState.Buttons & ovrTouch_X)
{
// Only submit the buffer if there is enough space available.
ovrHapticsPlaybackState playbackState;
ovrResult result = ovr_GetControllerVibrationState(Session,
ovrControllerType_LTouch, &playbackState);
Oculus Rift | Oculus Touch Controllers | 59
if (playbackState.RemainingQueueSpace >= bufferSize)
{
ovr_SubmitControllerVibration(Session, ovrControllerType_LTouch,
&bufferX);
}
}
if (inputState.Buttons & ovrTouch_Y)
{
// Only submit the buffer if there is enough space available.
ovrHapticsPlaybackState playbackState;
ovrResult result = ovr_GetControllerVibrationState(Session,
ovrControllerType_LTouch, &playbackState);
if (playbackState.RemainingQueueSpace >= bufferSize)
{
ovr_SubmitControllerVibration(Session, ovrControllerType_LTouch,
&bufferY);
}
}
// Just render a standard scene in the HMD, to keep the code as simple as
possible.
// The controllers are not rendered in the scene.
for (int eye = 0; eye < 2; ++eye)
{
XMMATRIX viewProj = Layer[0]->RenderSceneToEyeBuffer(MainCam, RoomScene,
eye);
}
Layer[0]->PrepareLayerHeader();
DistortAndPresent(1);
}
free(dataBufferX);
free(dataBufferY);
free(dataBufferA);
free(dataBufferBLeft);
free(dataBufferBRight);
}
};
};
Emulating Gamepad Input with Touch
Touch controllers can partially emulate Microsoft XInput API gamepad input without any code changes.
However, you must account for the missing logical and ergonomic equivalences between the two types of controllers.
How it Works
The Oculus runtime has a gamepad emulation mode to map Touch controller input to Microsoft XInput API input. This lets you allow users to use their Touch controllers in games designed for gamepad controllers without having to modify your code.
The Oculus runtime enables gamepad via Touch when all the following conditions are met:
• VR must have focus.
• The app must have the Gamepad Emulation Via Touch option enabled.
• The Touch controllers must be detected.
• A gamepad must not be detected. You cannot use Touch to emulate a second gamepad.
Touch controllers revert back to being Touch controllers when the app closes or when the user presses the
Oculus button and exits to Home.
60 | Oculus Touch Controllers | Oculus Rift
Activating and Deactivating Gamepad Emulation
Gamepad emulation is an Oculus Store attribute attached to the build binary. It can only be enabled or disabled by uploading a new build.
To set a gamepad emulation mode:
1. Upload a new Rift build from the Oculus Developer Dashboard.
2. On the build upload information page, select the appropriate
Gamepad Emulation Via Touch option:
• Off
• Twin Stick
• Left D-pad
• Right D-pad
Gamepad Emulator Compatibility Guidelines
If your gamepad control mechanism is affected by any of the following limitations, your game is probably a poor candidate for using gamepad emulation. Instead of emulation, consider creating a new control scheme for
Touch controllers to provide a better experience for your users.
Haptics
The current implementation does not include haptics. If vibration and haptics are critical to your experience, such as in a rhythm game, your app might not work well with emulation.
D-pad
The Touch controller does not have a D-pad. If your control scheme uses the D-pad and both sticks, your app might not work well with emulation.
If your control scheme only needs one stick, you can use our emulation modes to map the D-pad to either the right or left Touch.
View button
Twin stick emulation does not have a view button (⧉). If this button is critical to your control scheme, your app might not work well with twin-stick emulation mode.
The D-pad emulation modes do feature a mapping for this button.
Left Touch thumbstick can’t be used with the X or Y buttons
Any action that combines left Touch thumbstick movement with X or Y button presses will be difficult or impossible for users because these actions are controlled by the same thumb. If your control scheme requires such an action, your app might not work well with emulation.
Testing Gamepad Emulation
You can force a gamepad emulation mode in the Oculus Runtime for testing purposes.
To force gamepad emulation:
1. Add or modify the following DWORD value in the Windows registry:
[HKEY_CURRENT_USER\SOFTWARE\Oculus]
"GamepadEmulationMode"=dword:mode_enum where mode_enum is one of the following:
•
00000002 - Twin stick mode.
Oculus Rift | Oculus Touch Controllers | 61
•
00000003 - Right D-pad mode.
•
00000004 - Left D-pad mode.
2. Restart the Oculus runtime service. The service name is "OVRService" and the display name is "Oculus VR
Runtime Service".
To revert to normal behavior, delete the
GamepadEmulationMode value from the Windows registry and then restart OVRService.
Twin Stick Gamepad Emulation Mode
Maps the two gamepad sticks to the Touch thumbsticks. The D-pad and view button (⧉) are not mapped.
Gamepad Input vibration
D-pad left stick left bumper left trigger
Touch Equivalent
NONE
NONE left thumbstick left grip left trigger
62 | Oculus Touch Controllers | Oculus Rift
Gamepad Input view button (⧉)
Xbox button menu button
(☰) right stick right bumper right trigger
A button
B button
X button
Y button
Touch Equivalent
NONE
Oculus button menu button
(☰) right thumbstick right grip right trigger
A button
B button
X button
Y button
Left D-Pad Gamepad Emulation Mode
Same as twin stick gamepad emulation mode except for the following:
Gamepad Input
D-pad left stick view button (⧉)
Touch Equivalent left thumbstick
NONE left thumbstick press
Right D-Pad Gamepad Emulation Mode
Same as twin stick gamepad emulation mode except for the following:
Oculus Rift | Oculus Touch Controllers | 63
Gamepad Input
D-pad right stick view button (⧉)
Touch Equivalent right thumbstick
NONE right thumbstick press
64 | SDK Samples and Gamepad Usage | Oculus Rift
SDK Samples and Gamepad Usage
Some of the Oculus SDK samples use gamepad controllers to enable movement around the virtual world.
This section describes the devices that are currently supported and setup instructions.
Xbox 360 Wired Controller for Windows
To set up the controller:
• Plug the device into a USB port. Windows should recognize the controller and install any necessary drivers automatically.
Logitech F710 Wireless Gamepad
To set up the gamepad for Windows:
1. Put the controller into ‘XInput’ mode by moving the switch on the front of the controller to the ‘X’ position.
2. Press a button on the controller so that the green LED next to the ‘Mode’ button begins to flash.
3. Plug the USB receiver into the PC while the LED is flashing.
4. Windows should recognize the controller and install any necessary drivers automatically.
Oculus Rift | Optimizing Your Application | 65
Optimizing Your Application
To provide the best user experience, your application must meet or exceed the minimum requirements to be considered for publication on the Oculus Store.
For more information about publishing requirements, see our
Publishing
documentation.
This section describes how to use tools we provide to optimize the performance of your application.
SDK Performance Statistics
The SDK performance statistics provide information about application and compositor performance on the system.
Stats are populated after each call to ovr_SubmitFrame. To get performance stats, use ovr_GetPerfStats().
To reset the stats, use ovr_ResetPerfStats.
Note: If your performance tool runs separately from your application, make sure to read the
VisibleProcessId in case your application loses focus.
The following table describes performance statistics:
Table 1: Statistics
Statistics
Description ovrPerfStatsPerCompositorFrame
AnyFrameStatsDropped
AdaptiveGpuPerformanceScale
The per-compositor frame statistics in the following tables.
If the app calls ovr_SubmitFrame at a rate less than 18 fps, then when calling ovr_GetPerfStats, expect AnyFrameStatsDropped to become ovrTrue while FrameStatsCount is equal to ovrMaxProvidedFrameStats.
An edge-filtered value that you can use to adjust the graphics quality of the application to keep the
GPU utilization in check. The value is calculated as:
(desired_GPU_utilization / current_GPU_utilization).
When this value is 1.0, the GPU is doing the right amount of work for the app. Lower values mean the application needs to reduce the GPU utilization.
Note: If the app directly drives render-target resolution using this value, make sure to take the square-root of the value before scaling the resolution with it.
Changing the render target resolution is only one of the many things your application can do increase or decrease the amount of GPU utilization. Since
66 | Optimizing Your Application | Oculus Rift
Statistics
Description
AswIsAvailable
AdaptiveGpuPerformanceScale is edge-filtered and does not change rapidly (i.e., it reports non-1.0 values once every couple of seconds), your application can make the necessary adjustments and continue watching the value to see if it has been satisfied.
Returns true if ASW is available for this system, based on the user's GPU, operating system, and debug override settings.
The following table describes statistics specific to your application's performance:
Table 2: Application Statistics
Statistic
Description
AppFrameIndex
AppDroppedFrameCount
AppMotionToPhotonLatency
AppQueueAheadTime
AppCpuElapsedTime
Index that increments with each ovr_SubmitFrame call.
Increments each time the application fails to submit a new set of layers using ovr_SubmitFrame() before the compositor is executed before each V-Sync (Vertical
Synchronization).
Latency from when the last predicted tracking information was queried by the application using ovr_GetTrackingState() to when the middle scanline of the target frame is illuminated on the HMD display.
This is the same information provided by the Latency
Timing HUD.
To improve CPU and GPU parallelism and increase the amount of time that the GPU has to process a frame, the SDK automatically applies queue ahead up to 1 frame. This value displays the amount of queue ahead time being applied (in milliseconds). For more information about adaptive queue ahead, see
on page 30.
The time difference from when the application continued execution on CPU after ovr_SubmitFrame() returned the subsequent call to ovr_SubmitFrame().
This will show "N/A" if the latency tester is not functioning as expected (e.g., HMD display is sleeping due to prolonged inactivity). This includes the IPC call overhead to the compositor after ovr_SubmitFrame() is called by the client application.
AppGpuElapsedTime
The total GPU time spent on rendering by the client application. This includes the work done by the application after returning from ovr_SubmitFrame(), using the mirror texture if applicable.
It can also includes GPU command-buffer "bubbles" if the application's CPU thread doesn't push data to
Oculus Rift | Optimizing Your Application | 67
Statistic
Description the GPU fast enough to keep it occupied. Similarly, if the app pushes the GPU close to full-utilization, the work on next frame (N+1) might be preempted by the compositor's render work on the current frame
(N). Because of how the application GPU timing query operates, this can lead to artificially inflated application GPU times as they will start to include the compositor GPU usage times.
The compositor operates asynchronously and will increment for each vsync, regardless of whether the application calls ovr_SubmitFrame.
The following table describes compositor statistics:
Table 3: Compositor Statistics
Statistic
Description
CompositorFrameIndex
CompositorDroppedFrameCount
CompositorLatency
CompositorCpuElapsedTime
CompositorGpuElapsedTime
CompositorCpuStartToGpuEndElapsedTime
CompositorGpuEndToVsyncElapsedTime
Index that increments each time the SDK compositor completes a distortion/TimeWarp pass.
Increments each time the compositor fails to present a new rendered frame at V-Sync (Vertical
Synchronization).
Specifies the TimeWarp latency, which corrects app latency and dropped frames.
The amount of time in seconds spent on the GPU by the SDK compositor. Any time spent on the compositor takes available GPU time away from the application.
The amount of time the GPU spends executing the compositor renderer. This includes TimeWarp and distortion of all layers submitted by the application.
The amount of active layers, their resolutions, and the requested sampling quality can all affect the GPU times.
The amount of time from when the CPU kicks off the compositor to when the compositor completes distortion and TimeWarp on the GPU. If the time is not available, it returns -1.0f.
The amount of time between when the GPU completes the compositor rendering to the point in time when V-Sync is hit and that buffer starts scanning out on the HMD.
The Asynchronous SpaceWarp (ASW) HUD displays activity and tracking statistics for ASW, which runs as part of the Oculus Runtime Compositor. ASW automatically activates when an application fails to meet the required native frame rate for the connected HMD. Once active, ASW forces the application to run at half the normal frame rate while extrapolating every other frame. This gives the application more time to complete its work.
68 | Optimizing Your Application | Oculus Rift
The following table describes Asynchronous SpaceWarp (ASW) statistics:
Table 4: ASW Statistics
Statistic
Description
AswIsActive
AswActivatedToggleCount
AswPresentedFrameCount
AswFailedFrameCount
Shows the availability and current status of ASW.
"Not Available" can be due to the OS and/or GPU type used on the PC. "Available - Not Active" will mean the application is successfully maintaining the required native refresh rate, so ASW is not activated.
Tracks the number of times ASW has been activated for the lifetime of the HMD.
Tracks the number of frames extrapolated by ASW that were displayed. When ASW is active, since the app is forced to run at half-rate, expect this value to increase by 45 fps on a 90 Hz refresh rate HMD.
Tracks the number of extrapolated frames ASW needed to display, but failed to prepare in time.
This can occur for different reasons, but might be caused by contention for OS resources or when the capabilities of the system are exceeded.
Oculus Debug Tool
The Oculus Debug Tool enables you to view performance or debugging information within your game or experience.
To use the tool:
1. Make sure that you have admin privilege. This is required in order to run the Oculus Debug Tool.
2. Go to Program Files\Oculus\Support\oculus-diagnostics\OculusDebugTool.exe. Note that the Oculus
Debug Tool should always be run directly from this location, in order to ensure a version match with the Oculus distribution. If you copy the Oculus Debug Tool to another location, it might not work after subsequent Oculus updates.
3. Double-click
OculusDebugTool.exe. The Oculus Debug Tool opens.
4. Select the Visible HUD to view. Options include: None (no HUD is displayed), Performance HUD, Stereo
Debug HUD, or Layer HUD.
5. If you selected Performance HUD, select which Performance HUD you want to view. Options include:
Latency Timing, Render Timing, Performance Headroom, and Version Information. For more information, see
on page 72.
The following is an example of the Performance HUD:
Oculus Rift | Optimizing Your Application | 69
6. If you selected Stereo Debug HUD, configure the mode, size, position, and color from the Stereo Debug
HUD options.
The following is an example of the Stereo Debug HUD:
70 | Optimizing Your Application | Oculus Rift
7. If you selected Layer HUD. select the layer for which to show information or select the Show All check box.
The following is an example of the Layer HUD:
Oculus Rift | Optimizing Your Application | 71
8. Select Launch App from the File menu and select the executable of the application.
9. Put on the headset and view the results.
Performance Profiler
The Oculus Performance Profiler displays a graph that shows statistics on the performance of your application.
You can view the statistics in real time or export them to a comma-separated value (CSV) file that you can import into a spreadsheet or database.
Note: For more information about the statistics collected, see
72.
To use the tool:
1. Go to Program Files\Oculus\Support\oculus-diagnostics\OculusDebugTool.exe.
2. Double-click
OculusDebugTool.exe. The Oculus Debug Tool opens.
3. Select Performance Profiler from the Mode list box.
4. Select Launch App from the File menu and select the application to profile.
5. Click Start.
6. Have someone put on the headset and start using with the application.
72 | Optimizing Your Application | Oculus Rift
7. View the results on screen.
Figure 13: Peformance Profiler
8. When you are finished, click Stop.
9. To export the results to CSV, click Export data to CSV and save the file.
Performance Head-Up Display
The Performance Head-Up Display (HUD) enables you or your users to view performance information for applications built with the SDK.
The Performance HUD screens are rendered by the compositor, which enables them to be displayed with a single SDK call to ovr_SubmitFrame. In the Oculus Debug Tool or OculusWorldDemo, you can toggle through the Performance HUD screens by pressing F11.
Performance Summary
The Performance Summary HUD displays the frame rate of the application and the unused hardware performance available. This HUD can be used by you or the user to tune an application's simulation and graphics fidelity. Because the user cannot disable V-Sync, it can be used to gauge performance instead of a frame rate counter. It is also useful for troubleshooting whether issues are related to the application or the hardware setup.
The following image shows the Performance Summary HUD:
Figure 14: Performance Summary HUD
Oculus Rift | Optimizing Your Application | 73
The following table describes each metric:
Metric Description
App Motion-to-Photon Latency Latency from when the last predicted tracking information was queried by the application using ovr_GetTrackingState() to when the middle scanline of the target frame is illuminated on the HMD display. This is the same information provided by the Latency Timing HUD.
Unused performance
Designed to help the user verify that the PC is powerful enough to avoid dropping frames, this displays the percentage of available PC performance not used by the application and compositor. This is calculated using the
CPU and GPU time tracked by the Application Render Timing HUD divided by the native frame time (inverse of refresh rate) of the HMD.
Note: As GPU utilization approaches 100%, adaptive queue ahead will choose an earlier render start point. If this start point overlaps with the compositor process in the previous frame, the performance will appear spiky. If you start to lower utilization, the graph will show an initial drop before becoming more linear.
Application Frames Dropped
Compositor Frames Dropped
Increments each time the application fails to submit a new set of layers using ovr_SubmitFrame() before the compositor is executed before each
V-Sync (Vertical Synchronization). This is identical to App Missed Submit
Count in the Application Render Timing HUD.
Increments each time the compositor fails to present a new rendered frame at V-Sync (Vertical Synchronization). This is identical to Compositor Missed
V-Sync Count in the Compositor Render Timing HUD.
74 | Optimizing Your Application | Oculus Rift
Latency Timing
The Latency Timing HUD displays the App to Mid - Photon, Timewarp to Photon - Start, and Timewarp to
Photon - Start graphs.
The following screenshot shows the Latency Timing HUD:
Figure 15: Latency Timing
The following table describes each metric:
Table 5: Latency Timing HUD
Metric
App Tracking to Mid-Photon
Timewarp to Mid-Photon
Flip to Photon - Start
Description
Latency from when the app called ovr_GetTrackingState() to when the target frame eventually was shown (i.e.illuminated) on the HMD display averaged mid - point illumination
Latency from when the last predictied tracking info is fed to the GPU for timewarp execution to the point when the middle scanline of the target frame is illuminated on the HMD display
Time difference from the point the back buffer is presented to the HMD to the point the target frame's first scanline is illuminated on the HMD display
Application Render Timing
The Application Render Timing HUD displays application-specific render timing information.
The following screenshot shows the Application Render Timing HUD:
Figure 16: Application Render Timing
Oculus Rift | Optimizing Your Application | 75
The following table describes each metric:
Table 6: Application Render Timing HUD
Metric Description
App Missed Submit Count
App Frame-rate
Increments each time the application fails to submit a new set of layers using ovr_SubmitFrame() before the compositor is executed and before each V-Sync
(Vertical Synchronization).
The rate at which application rendering calls ovr_SubmitFrame(). It will never exceed the native refresh rate of the HMD as the call to ovr_SubmitFrame() throttles the application's CPU execution as needed.
App Render GPU Time
The total GPU time spent on rendering by the client application. This includes the work done by the application after returning from ovr_SubmitFrame(), using the mirror texture if applicable.
It can also includes GPU command-buffer "bubbles" if the application's
CPU thread doesn't push data to the GPU fast enough to keep it occupied.
Similarly, if the app pushes the GPU close to full-utilization, the work on next frame (N+1) might be preempted by the compositor's render work on the current frame (N). Because of how the application GPU timing query operates, this can lead to artificially inflated application GPU times as they will start to include the compositor GPU usage times.
App Render CPU Time The time difference from when the application continued execution on CPU after ovr_SubmitFrame() returned the subsequent call to ovr_SubmitFrame().
This will show "N/A" if the latency tester is not functioning as expected (e.g.,
HMD display is sleeping due to prolonged inactivity). This includes the IPC
76 | Optimizing Your Application | Oculus Rift
Metric
App Queue Ahead Time
Description call overhead to the compositor after ovr_SubmitFrame() is called by the client application.
To improve CPU and GPU parallelism and increase the amount of time that the GPU has to process a frame, the SDK automatically applies queue ahead up to 1 frame. This value displays the amount of queue ahead time being applied (in milliseconds). For more information about adaptive queue ahead,
on page 30.
Compositor Render Timing
The Compositor Render Timing HUD displays render timing information for the Oculus Runtime Compositor.
The Oculus Compositor applies distortion and TimeWarp to the layered eye textures provided by the VR application.
The following screenshot shows the Compositor Render Timing HUD:
Figure 17: Render Timing
The following table describes each metric:
Table 7: Compositor Render Timing HUD
Metric Description
Compositor Missed V-Sync Count Increments each time the compositor fails to present a new rendered frame at V-Sync (Vertical Synchronization).
Compositor Frame-rate The rate of the final composition; this is independent of the client application rendering rate. Because the compositor is always locked to
V-Sync, this value will never exceed the native HMD refresh rate. But, if
Oculus Rift | Optimizing Your Application | 77
Metric
Compositor GPU Time
Comp Gpu-End to V-Sync
Description the compositor fails to finish new frames on time, it can drop below the native refresh rate.
The amount of time the GPU spends executing the compositor renderer.
This includes TimeWarp and distortion of all layers submitted by the application. The amount of active layers, their resolutions, and the requested sampling quality can all affect the GPU times.
The amount of time between when the GPU completes the compositor rendering to the point in time when V-Sync is hit and that buffer starts scanning out on the HMD.
Asynchronous SpaceWarp Stats
The Asynchronous SpaceWarp (ASW) HUD displays activity and tracking statistics for ASW, which runs as part of the Oculus Runtime Compositor. ASW automatically activates when an application fails to meet the required native frame rate for the connected HMD. Once active, ASW forces the application to run at half the normal frame rate while extrapolating every other frame. This gives the application more time to complete its work.
The following screenshot shows the ASW HUD:
Figure 18: ASW Stats
The following table describes each metric:
Table 8: ASW Stats HUD
Metric
ASW Status
Description
Shows the availability and current status of ASW. "Not Available" can be due to the OS and/or GPU type used on the PC. "Available - Not Active" will
78 | Optimizing Your Application | Oculus Rift
Metric Description mean the application is successfully maintaining the required native refresh rate, so ASW is not activated.
ASW Active-Toggle Count
ASW Presented-Frame Count Tracks the number of frames extrapolated by ASW that were displayed. When
ASW is active, since the app is forced to run at half-rate, expect this value to increase by 45 fps on a 90 Hz refresh rate HMD.
ASW Failed-Frame Count
Tracks the number of times ASW has been activated for the lifetime of the
HMD.
Tracks the number of extrapolated frames ASW needed to display, but failed to prepare in time. This can occur for different reasons, but might be caused by contention for OS resources or when the capabilities of the system are exceeded.
Version Information
The Version Information HUD displays information about the HMD and the version of the SDK used to create the app.
The following screenshot shows the Version Information HUD:
Figure 19: Version Info HUD
The following table describes each piece of information:
Name
OVR SDK Runtime Ver
Description
Version of the currently installed runtime. Every VR application that uses the OVR SDK since 0.5.0 uses this runtime.
Oculus Rift | Optimizing Your Application | 79
Name
OVR SDK Client DLL Ver
HMD Type
HMD Serial
HMD Firmware
Sensor Serial
Sensor Firmware
Description
The SDK version that the client app was compiled against.
The type of HMD.
The serial number of the HMD.
The version of the installed HMD firmware.
The serial number of the positional sensor.
The version of the installed positional sensor firmware.
Performance Indicator
Asynchornous TimeWarp (ATW) can mask latency and judder issues that would normally be apparent. To help you identify when your application or experience isn't performing and to test your game or experience before submitting it, Oculus provides performance indicators.
When enabled, a letter code appears in the upper right of the headset whenever the application is experiencing a performance issue. The following figure shows an example of a performance indicator with L and F displayed:
Figure 20: Performance Indicator
To enable the performance indicator, set the following registry key:
HKLM\SOFTWARE\Oculus VR, LLC\LibOVR
FrameDropHUDEnabled (DWORD)
Enable = 1
Disable = 0
80 | Optimizing Your Application | Oculus Rift
The performance indicator can return the following codes:
• L—a latency issue is occurring; more than one frame of queue ahead is being applied.
• F—the application is not maintaining frame rate.
• C—the compositor is not maintaining frame rate. Possible causes include:
• Programs, such as anti-virus software, are overloading the CPU.
• The CPU cannot handle the number of threads.
• The CPU or GPU does not meet the recommended specification.
• Certain patterns of GPU usage, such as heavy usage of shaders and tessellation, are affecting frame rate.
• There is an issue with the GPU driver.
• There is an unknown hardware issue.
• U—an unknown error occurred.
Each warning lasts one frame. So, if L stays visible, the application is having continuous latency issues.
Compositor Mirror
The Compositor Mirror tool displays the content that appears within the Rift headset on your computer monitor. It has several display options that are useful for development, troubleshooting, and presentations.
Anything that appears in the Rift headset can be shown in the Compositor Mirror tool, including Oculus Home,
Guardian System boundaries, in-game notifications, and transition fades. It is compatible with all games and experiences regardless whether they are developed using the native PC SDK or a game engine.
Finding the Tool
The Compositor Mirror tool is located in C:\Program Files\Oculus\Support\oculus-diagnostics\OculusMirror.exe
Image Display Options
The Compositor Mirror tool provides several display options.
The default mode is recommended for live presentations and demos. If you double-click OculusMirror from
Windows Explorer or run from the command line without specifying any options, the tool displays a 1366x768
(pixel) window showing a rectilinear view of the right eye image, along with the Guardian System boundary layer and the Oculus notification layer.
The other display modes are as follows:
OculusMirror.exe
--
LeftEyeOnly
Display the left eye image in rectilinear view.
Oculus Rift | Optimizing Your Application | 81
82 | Optimizing Your Application | Oculus Rift
this example, the user's hands have protruded through the
Guardian
System boundary in two locations, as indicated by the circular holes in the boundary.
OculusMirror.exe
--
RectilinearBothEyes
-both eye images
Display both in rectilinear view.
rectilinear view along with the
Guardian
System boundary.
In
Oculus Rift | Optimizing Your Application | 83
84 | Optimizing Your Application | Oculus Rift
OculusMirror.exe
--
PostDistortion other options that appear within the headset.
For example, the following output is produced with
--
Display both eye images corrected for lens distortion and chromatic aberration.
This will also display any
PostDistortion, but also shows the
Guardian system boundary, since it is visible within the headset.
No additional options are allowed
Oculus Rift | Optimizing Your Application | 85
86 | Optimizing Your Application | Oculus Rift
in conjunction with
--
PostDistortion.
For example, you cannot explicitly exclude notifications or the
Guardian system boundary, if they are visible within the headset.
Oculus Rift | Optimizing Your Application | 87
88 | Optimizing Your Application | Oculus Rift
Changing the Window Size
You can change the size of the window by dragging the window borders on the desktop. You can also use the
--Size width height command to set the size of the window (in pixels) from the command line. If you exceed the resolution of your main display, the window size is reduced to fit. For example:
OculusMirror.exe --Size 2160 2160 --RightEyeOnly
Displaying Notifications
You can display the Oculus notifications layer by using
--IncludeNotifications. For example:
OculusMirror.exe --RectilinearBothEyes --IncludeNotifications
Flash on Frame Drops
It is possible that the CompositorMirror tool may drop frames that are displayed within the Rift headset. If you wish to clearly see when this happens, use the
--FlashFrameDrops command. This will cause the display in the CompositorMirror to flash whenever a frame is dropped. For example:
OculusMirror.exe --FlashFrameDrops --RectilinearBothEyes
Oculus Rift | Pairing the Oculus Touch Controllers | 89
Pairing the Oculus Touch Controllers
After you receive your Touch Controllers, you need to pair them with the headset.
To pair your Touch controllers:
1. Make sure your headset is connected.
2. Launch the Oculus app.
3. Click the menu icon in the upper right to open the dropdown menu and select Settings.
4. Click Settings.
5. Click Devices.
6. Select Add Left Touch from Configure Rift and follow the on-screen instructions to pair the controller and update the firmware. When the process is finished, the Left Touch Paired screen displays.
7. Select Add Right Touch from Configure Rift and follow the on-screen instructions to pair the controller and update the firmware. When the process is finished, the Right Touch Paired screen displays.
Note: If the Add Left Touch and Add Right Touch options do not appear, please contact Developer
Relations.
90 | Asynchronous SpaceWarp | Oculus Rift
Asynchronous SpaceWarp
Asynchronous SpaceWarp (ASW) enables users to run the Oculus Rift on lower specification hardware than our current recommended specification.
Overview
ASW applies animation detection, camera translation, and head translation to previous frames in order to predict the next frame. As a result, motion is smoothed and applications can run on lower performance hardware.
The Rift operates at 90Hz. When an application fails to submit frames at 90Hz, the Rift runtime drops the application down to 45Hz with ASW providing each intermediate frame.
By default, ASW is enabled for all supported Rifts.
ASW tends to predict linear motion better than non-linear motion. If your application is dropping frames, you can either adjust the resolution or simply allow ASW to take over.
Requirements
ASW requires the following:
• Oculus Runtime 1.9 or later
• Windows 8 or later
• For Nvidia, driver 373.06 or later
• For AMD, driver 16.40.2311 or later
Until the minimum specification is released, we recommend the following GPUs for ASW testing:
Manufacturer
Nvidia
Nvidia
AMD
Series
Pascal
Maxwell
Polaris
Minimum RAM
3GB
4GB
4GB
Minimum Model
1060
960
470
Testing ASW
To enable ASW testing:
1. Open your registry editor.
2. Navigate to HKLM\Software\Oculus VR, LLC\LibOVR.
3. Create the DWORD
AswEnabled key and set it to 1.
While testing your application with ASW, you can switch between rendering modes:
• Control-Numpad1: Disables ASW and returns to the standard rendering mode.
• Control-Numpad2: Forces apps to 45Hz with ASW disabled. Depending on the application, you are likely to experience judder.
• Control-Numpad3: Forces apps to 45Hz with ASW enabled. Enabling and disabling ASW will help you see the effects of ASW.
• Control-Numpad4: Enables ASW. ASW automatically turns on and off, depending on whether the app maintains a 90Hz frame rate. This is the default runtime rendering mode.
Oculus Rift | Mixed Reality Capture Setup Guide | 91
Mixed Reality Capture Setup Guide
Mixed reality capture places real-world objects in VR. In other words, it mixes images from the real world with the virtual one.
Oculus mixed reality capture is designed for developers, marketers, and influencers to share an intimate VR experience with viewers. By placing yourself and others in a VR scene, you can more effectively demonstrate the interaction between the user and virtual world.
(Courtesy of Medium and artist Dominic Qwek - https://www.oculus.com/medium/ )
Note: Mixed reality capture is only available for select Rift applications.
How does mixed reality capture work?
Mixed reality capture uses an external camera to capture real-world images, against a green-screen, and combine them with a scene generated by a VR application to create a single blended scene. A user can then interact with the VR scene, and a video can be created that shows the user’s interactions in the real-world overlaid on the VR experience
92 | Mixed Reality Capture Setup Guide | Oculus Rift
Before You Begin
Before any calibrations, configuration, or implementation, you need to acquire a PC and camera that meet the
performance requirements of mixed reality capture. Review the
a list of PCs that support the additional performance requirements. These are not required hardware to use mixed reality capture.
Similarly, the
on page 94 page contains information about the room and sensor setup for use with mixed reality capture. These are not required configurations, but are provided as guidelines when setting up your room for mixed reality capture.
Review the guidelines for creating a safe use environment in the Health and Safety warnings located at
https:// www.oculus.com/legal/health-and-safety-warnings/
.
Getting Started
There are two steps to capturing mixed reality capture scenes. Setting up your external camera and a VR
Object, and compositing the final scene.
Setup
First, you need to decide if you’ll be using a static or dynamic camera to capture the scene. Static cameras will only give you a single unmovable view in the scene, while a dynamic camera, a camera that you've associated with a VR Object, allows you to move the camera around in the scene.
If you use a dynamic camera, you’ll start by creating the VR Object. This object, a Touch controller, tracks the
on page 97. You may skip this step if you’re planning to use a static camera.
The next step is to calibrate your external camera to capture the real world images. The calibration process removes image distortion and determines the depth and position of objects captured in the scene. Information
about calibrating the camera is found on the
on page 100 page.
Compositing
Once you’ve finished the setup and configuration, you can launch the mixed reality capture capable app and composite the final scene together. Please review the
Launch the App and Composite the Scene
103 page for information about this process.
Non-Oculus equipment/service references are not recommendations, guarantees, or endorsements of products or performance. Your results may vary. These are instead options to vary your own customized experiences.
Furthermore, incorrect installation or use of any products or items can damage your computer, space, or otherwise impact performance. If you’re not experienced or qualified to do so yourself, please seek assistance.
Compatible Systems
The following mixed reality capture PCs and cameras are capable of handling the additional requirements of mixed reality capture.
Compatible PCs
These systems are not requirements or recommendations, these are suggested configurations that can handle the added bandwidth requirements of mixed reality capture. Other systems and builds may work with your MR configuration.
Build Type
Oculus Ready Systems
Custom Builds
Oculus Rift | Mixed Reality Capture Setup Guide | 93
Suggested Configurations
Falcon Northwest Tiki (2017) - ASUS ROG STRIX
Z270-I
System is readily available for configuration and purchase from Falcon Northwest.
Suggested configuration:
• 1 sensor in ASMedia 2142 USB 3.1 back port
(Driver version 1.16.38.1 or higher).
• 1 sensor and 1 camera in Intel USB 2.0 back ports.
• 1 sensor and 1 HMD in the Intel USB 3.0 back ports.
• Mouse/Keyboard in 2 Intel USB 3.0 ports on case.
EVGA Z170 Classified K
This motherboard works well when paired with 16gb of RAM, SSDs for storage space, and a GTX 1080 or higher.
Suggested configuration:
• 1 sensor and 1 camera in back Intel USB 2.0 ports.
• 1 sensor in back ASMedia USB 3.1 port (driver version 1.16.38.1 or higher).
• 1 sensor and 1 HMD in back Intel USB 3.0 ports.
• Mouse and keyboard in back Intel USB 3.0 ports.
ASUS Z170 DELUXE
This motherboard works well when paired with 16gb of RAM, SSDs for storage space, and a GTX 1080 or higher.
Suggested configuration:
• 1 HMD in Intel USB 3.0 back port.
• 1 sensor in Intel USB 2.0 back port.
• 2 sensors in ASMedia 3.1 back ports (driver version
1.16.38.1 or higher).
• Mouse, keyboard, and camera plugged into case ports.
• Alternatively, if the case only has 2 Intel USB 3.0
ports, plug the mouse and keyboard into 2 of the
ASMedia 3.1 back ports and the webcam into one of the case ports.
ASUS Z97 Deluxe USB 3.1
This motherboard works well when paired with 16gb of RAM, SSDs for storage space, and a GTX 1080 or higher.
Suggested configuration:
94 | Mixed Reality Capture Setup Guide | Oculus Rift
Build Type
Expansion Cards
Suggested Configurations
• 2 sensors in ASMedia 3.1 back ports (driver version
1.16.38.1 or higher).
• 1 sensor and 1 HMD in Intel USB 3.0 back ports.
• Mouse, keyboard and camera in Intel USB 2.0 back ports.
Startech Renesas 4 Port PCI Express (PCIe)
SuperSpeed USB 3.0 Card Adapter w/ 4 Dedicated
5Gbps Channels
• Works well when partnered with an Intel 100
Series, 200 Series or 9 Series motherboard.
• Avoid the X99 Deluxe and X99 Deluxe II due to sensor tracking issues with onboard WiFi chip.
• 3 sensors and HMD can be plugged into card.
• Performance could potentially be limited if being used in a system with a subpar motherboard or low end CPU due to PCI issues.
• Camera and mouse/keyboard can be plugged into the back motherboard ports without issue.
Compatible Cameras
Our camera calibration tool, introduced on the
on page 100 page, can support any
USB camera. However, cameras with higher fidelity and performance will result in higher-quality mixed reality capture scenes. HDMI cameras are supported if you connect them to the PC using a HDMI to USB dongle and plugging into one of the USB ports as described above.
Room and Sensor Setup
Mixed reality captures images from an external camera that you'll compose into the virtual app that you're using. This guide will review some room configurations that Oculus used when developing mixed reality; your use-case and implementation may be different.
Note: 4 sensor configurations are not supported by Oculus. However, 4 sensor configurations may offer high-performance mixed reality tracking when used in combination with the 4 USB expansion card listed
Green Screen Setup
Guides reviewing materials to use and how to construct a green screen are readily available online.
Sensor Configuration
The following sensor setups are starting points for your configuration. The optimal sensor setup may vary depending on your room configuration and app you're using. When setting up your sensors, it is important that you do not occlude the sensors. Occlusion could lead to a loss of tracking.
3 Sensor Configurations
Oculus Rift | Mixed Reality Capture Setup Guide | 95
When using 3 sensor configurations, you need to make sure that you do not occlude more than 1 sensor at any time. The following are examples of 3 sensor configurations.
Configuration one:
Configuration two:
96 | Mixed Reality Capture Setup Guide | Oculus Rift
4 Sensor Configurations
4 sensor configurations have shown to be reliable configurations for mixed reality capture. The following is an example of a 4 sensor configuration.
Oculus Rift | Mixed Reality Capture Setup Guide | 97
VR Object Setup
Mixed reality capture uses a VR Object to track the location of a dynamic camera. This page describes the process of setting up a VR Object. If you’re planning to use a static camera that does not move around the scene, you may skip this step.
A VR Object is a Touch controller with updated firmware. You may use one of the existing Touch controllers paired to your HMD as a VR Object. However, you’ll need to re-pair the controller to your HMD if you want to use it as a regular Touch controller again. We recommend using a third Touch controller that can be a permanent VR Object.
98 | Mixed Reality Capture Setup Guide | Oculus Rift
Note: Early, pre-production, Touch controllers (version C06x and earlier) will not work as VR Objects.
Identify these controllers by checking the serial number found inside the battery housing. These Touch controllers will have the following serial number: "WMTCL1xxxxxxx" or "WMTCR1xxxxxxx" for left and right Touch, respectively.
To create your VR Object you’ll need to update the device firmware.
1. Launch the Oculus desktop application, open the settings menu, and select the Devices tab. Then, from the dropdown, select “Pair VR Object (Advanced)”.
Oculus Rift | Mixed Reality Capture Setup Guide | 99
2. On the first screen, identify which controller, left or right, you are using as the VR Object.
3. Follow the on-screen prompt to pair the controller as a VR Object. This process will update the device firmware.
The VR Object will remain active for 24 hours while the proximity sensor on the HMD is covered. Otherwise, the
VR Object will go into standby mode after 30 seconds. Any button press or movement will return the VR Object to active mode.
To re-pair your Touch controller to the original HMD (if you did not use a 3rd Touch).
1. In the same dropdown that you selected the option to pair your VR Object, select ‘Pair Touch’.
2. Follow the on-screen prompts to pair the controllers.
Mounting the VR Object
Once you've created your VR Object, you'll need to attach the Touch to your external camera. There are a number of ways to do this, we recommend using the mount provided in the
Mixed Reality Capture Mount
Accessory Guidelines
. The download contains two CAD files that can be used to 3D print a camera mount that holds a Touch controller, one for each a right and left Touch controller. The mount is designed to be used with webcams and small DSLRs. Larger cameras may not fit on the mount.
Print the Mount
First, print the controller mount on a 3D printer. For best results, print the mount on the highest quality setting using Polyjet or SLS print methods. We have found that ABS, or equivalent, print materials work well with this mount design.
Mount Configuration and Assembly
Once you've printed the mount, you can attach the camera, VR Object, and tripod (or a steady camera mount).
1. Use a 3/4 inch bolt or cap screw to attach the mount to the tripod. A nut or wing nut may be used to secure the mount for a stable connection.
2. Use a 1/2 inch bolt or cap screw with a washer to attach the camera to the mount. If a washer is not used, you may need to use a shorter bolt depending on the camera you're using.
3. (Optional) You may wish to use small zip ties to secure the Vr Object to the mount. A change in relationship between the VR Object and camera may require you to reclibrate the camera.
4. The following image shows the complete mount using a small DSLR.
100 | Mixed Reality Capture Setup Guide | Oculus Rift
Camera Calibration
Calibrate your external camera for mixed reality capture using the provided CameraTool.
This guide will walk you through the steps to capture information about your camera and remove any distortion present when creating a mixed reality capture scene. The CameraTool gives you the option to configure a static camera or a dynamic camera that can be moved around the scene. This guide assumes that you have previously completed the Rift and Touch setup. For best results we recommend that you have your room set up in the configuration that you will use for mixed reality capture.
Oculus Rift | Mixed Reality Capture Setup Guide | 101
Before You Begin
Before you launch the camera calibration process, please complete the following steps:
1. Review the room and sensor setup information on
during the camera calibration process that the VR Object (if using) is within view of at least 2 sensors. The following is an example of how the room could be configured while calibrating the camera:
2. Review the recommended configurations on the
how the sensor, camera, and HMD are configured in the USB ports.
3. Print out the image provided with the CameraTool (C:\Program Files\Oculus\Support\oculus-diagnostics
\CameraCalibrationTool\Assets\pattern.png). We recommend adhering the pattern to a flat piece of cardboard to keep the pattern flat during the calibration process.
102 | Mixed Reality Capture Setup Guide | Oculus Rift
4. Cover the proximity sensor in your HMD with a sticker or piece of tape, you need to keep this sensor covered throughout the duration of the camera calibration process. The proximity sensor is located between the eye cups at the top of the HMD.
Calibrate The External Camera
In this process, you'll run the CameraTool and move a printout and your HMD around in several orientations to capture information about your camera.
To configure your camera for mixed reality capture, follow the steps below:
1. Launch CameraTool.exe (C:\Program Files\Oculus\Support\oculus-diagnostics\CameraCalibrationTool\).
Select 'Calibrate Camera'. If you plan to use a dynamic camera with a VR Object, make sure the VR Object is active, in view of the sensors, and attached to the external camera when you launch the tool.
2. Select your camera from the dropdown list of available cameras.
3. Enter a name for the camera configuration. You'll use this to later identify this configuration.
4. Enter the camera's resolution. This information is available from the camera manufacturer.
5. Select the prompt on screen to 'Start Intrinsic Calibration'. Align your checkerboard printout with the box placed on the screen, fit the pattern to the size of the box by adjusting the distance of the pattern from the camera. When you’ve lined up the image of the pattern in the box, press the X or A button on a Touch controller, or the button in the CameraTool window. Once complete, the calibration tool will use those images to remove any camera distortion. You’ll receive a reprojection error value indicating the quality of the configuration. Values <1 are acceptable, anything higher and we recommend that you run the calibration again.
Note: You may notice pincushioning around the edges of the camera output after this step, this is normal and the result of distortion removal.
6. Choose your tracked object (either a Touch controller or a VR Object), or select that this will be a Static
Camera. If using a VR Object, you need to complete the process of updating the firmware before running the CameraTool.
7. Position your Rift in the boxes that appear on the screen. You’ll need to adjust the distance of the headset from the camera to get the image of the Rift the same size as the box on the screen. It is important in this step to align the front face of the Rift as precisely as possible in the rectangles on the screen. When you’ve lined up the image of the Rift in the box, press the X or A button on a Touch controller, or the button in the
CameraTool window.
8. Finally, virtual controllers will be overlaid on your Touch controllers to demonstrate the result of your calibration. You may use sliders provided in the CameraTool to adjust the configuration if you wish. You’ll want to make sure the controllers are completely still for this step. The input lag from the camera makes the process difficult if the Touch controllers are moving.
Once you’ve completed the calibration, you can save the configuration to the system or a file. Saving to the system will use this calibration as long as the Oculus desktop app is running. Saving to a file will allow you to reload this configuration at any time in the future. To reload a configuration, launch the CameraTool and select
'Load Camera File'. Exit the CameraTool when you're finished saving the files.
Troubleshoot Your Camera Calibration
Sometimes things don't work as expected after the camera calibration process. This section will help you through some of the common issues experienced during the calibration process.
1.
My camera is lagging or jittery.
a. Check to see how you have your sensors and external camera configured in the USB ports. These are all high-bandwidth devices and can overload a chipset. Review how your devices are distributed in the
Windows Device Manager and rearrange if necessary.
Oculus Rift | Mixed Reality Capture Setup Guide | 103
b. Using a USB 3.0 camera with a USB 2.0 extension cable, or a USB 3.0 extension cable over 10 feet, may introduce camera lag.
c. Finally, if the previous two steps did not resolve the issues with lag, consider using the USB expansion
2.
I've completed the camera calibration process, but when I move my VR Object and webcam the Touch controllers aren't aligned properly with their virtual representation. Please review step 2 in the 'Before
You Begin' section above. Make sure your sensors can see the ring around the Touch controller you're using as the VR object throughout the entire process.
3.
I keep getting a high score (>1) for my camera calibration. During the process, make sure that you fit the entire checkerboard pattern into the rectangle. Changing the rotation of the image as well as changing the angle of the checkerboard slightly with each image.
4.
The CameraTool is detecting my camera but showing a black screen in the preview window. Make sure that another application isn't trying to use the webcam. The CameraTool needs to be the primary user of the webcam during calibration.
Launch the App and Composite the Scene
This page describes the process of launching the app and rendering the final mixed reality capture scene.
There are two ways that apps may choose to render mixed reality capture scenes. One way is that the app may handle all compositing, green screen clipping, and composition internally. The other is to use a 3rd party application, like
OBS Studio
or
XSplit
, to compose the scene.
External Compositing with a 3rd Party App
If the app you're using requires you to composite the app externally, this section of the guide describes the process using
OBS Studio
as an example. Other programs, such as
XSplit
, can be used for this process as well.
First, if you have not done so already, run the CameraTool to calibrate the camera or load a configuration file.
This process needs to be repeated each time you launch Oculus to use a mixed reality capture compatible app.
Information about this process can be found on the Camera Calibration page.
Next, launch the app in mixed reality capture mode. How you launch mixed reality capture mode will vary depending on how the mode has been implemented in the app. Please refer to the app's instructions for information about launching in mixed reality capture mode.
Then, in OBS Studio you can start capturing the game and external camera output.
1. Create your first source. This will be the background of the scene, so we'll use 'Window Capture' to grab the entire mirrored window. Select the window that contains the scene from the VR app.
2. If the app also captures a foreground of the scene, the output window may contain 2 images, one foreground and one background. Add a filter [Crop/Pad] to only select the half window and associate that with the foreground and background captures. The foreground will also contain the chroma key that you'll clip from the final scene.
3. Fill both the background and foreground to the whole compositing window by selecting Transform
→
Stretch to Screen.
4. Create another source for your external camera and apply the same transform that you did to the foreground and background images.
5. Add the chroma key effect filter to remove the green-screen from your external camera and foreground captures.
6. Make any adjustments to the capture windows or chroma keys required. The order of the layers should be foreground, external camera image, then background.
104 | Mixed Reality Capture Setup Guide | Oculus Rift
Troubleshooting Your External Compositing
Sometimes things don’t work how you expect. This section will review some of the common issues when compositing externally.
1.
When I try to overlay the OWD frames in OBS, they aren't showing up properly. Make sure that you're using the same resolution in OBS when cropping as the resolution you set in the Camera Calibration tool.
Also, uncheck the 'Relative' box in the crop/pad filter.
2.
My camera lagging behind with my Touch controllers in OBS. The VR app renders content faster than your external camera. Add a 'Render Delay' filter to your foreground and background until the images line up.
3.
OBS isn't picking up my webcam after I set everything up in the mixed reality capture app. Make sure that when you finish the CameraTool calibration, first save the calibration to an XML file, then save the camera to OVRserver, and then close the Camera Tool before opening OBS.
Internal Direct Compositing
The app you're using may composite the scene for you by retrieving the input from the external camera and mixing the images with the virtual scene from the app. In this case you'll need to refer to the in-app prompts about how to set up the scene when you launch the app in mixed reality capture mode.
That's it. You've completed the mixed reality capture setup and are ready to share your experiences.
Shkarko
Lidhja publike u përditësua
Lidhja publike për bisedën tuaj është përditësuar. You may view and manage your chat links after registration here.
Lidhja publike e dokumentit
Reklamë
