This document describes support for creating a digital representation of the observed environment and estimating in real-time the camera pose by leveraging the 3D camera.
This document was published by the Crosswalk Project as an API Draft. If you wish to make comments regarding this document, please send them to crosswalk-dev@lists.crosswalk-project.org. All comments are welcome.
ScenePerception
The ScenePerception interface provides methods to
track and scan scenes for augmented reality applications.
The ScenePerception interface is exposed through
realsense module.
init get called and successfully returned, otherwise it will fail with error.
init to re-initialize it if you want to use again.
init is called.
[R | T] = [Rotation Matrix | Translation Vector]
where R = [ r11 r12 r13 ]
[ r21 r22 r23 ]
[ r31 r32 r33 ]
T = [ tx ty tz ]
Camera pose sequense layout should be: [r11 r12 r13 tx r21 r22 r23 ty r31 r32 r33 tz]
This method returns a promise.
The promise will be fulfilled with the VolumePreviewData instance if there are no errors.
The promise will be rejected with the DOMException object if there is a failure.
[R | T] = [Rotation Matrix | Translation Vector]
where R = [ r11 r12 r13 ]
[ r21 r22 r23 ]
[ r31 r32 r33 ]
T = [ tx ty tz ]
Camera pose sequense layout should be: [r11 r12 r13 tx r21 r22 r23 ty r31 r32 r33 tz]
This method returns a promise.
The promise will be fulfilled with the CameraIntrinsics instance if there are no errors.
The promise will be rejected with the error message if there is a failure.
The getSurfaceVoxels function exports the centers of the voxels intersected by the surface scanned.
The voxels size is set based on the resolution of the color images.
Optionally, you can specify the region of interested bounding box.
The getSurfaceVoxels function returns promise.
The promise will be fulfilled with surface voxels in batches
if there are no errors, the dataPending attribute in the result will be true when there are remaining surface voxels.
Please call the function again until the dataPending flag returns false.
The promise will be rejected if there is a failure.
If useColor was set to false by calling configureSurfaceVoxelsData function before
getSurfaceVoxels function, the surfaceVoxelsColor member of
the returned result from getSurfaceVoxels function will be null.
Save the mesh data of the volume to an ASCII file. It will return a blob with 'text/plain' type.
getMeshData(). Set the thresholds indicating the magnitude of changes occurring in any block that would be considered significant for re-meshing.
[R | T] = [Rotation Matrix | Translation Vector]
where R = [ r11 r12 r13 ]
[ r21 r22 r23 ]
[ r31 r32 r33 ]
T = [ tx ty tz ]
[r11 r12 r13 tx r21 r22 r23 ty r31 r32 r33 tz]
This interface sets the configurations of getSurfaceVoxels function.
setMeshingRegion function sets the meshing region of interest for the getMeshData function.
clearMeshingRegion function removes any previously set meshing region of insterest for the getMeshData function.
A property used to set the EventHandler (described in [[!HTML]])
for the CheckingEvent that is dispatched
to ScenePerception when a frame is checked.
A property used to set the EventHandler (described in [[!HTML]])
for the ErrorEvent that is dispatched
to ScenePerception when there is an error.
A property used to set the EventHandler (described in [[!HTML]])
for the Event that is dispatched
to ScenePerception when mesh is updated.
A property used to set the EventHandler (described in [[!HTML]])
for the SampleProcessedEvent that is dispatched
to ScenePerception when mesh is updated.
CheckingEvent
a positive value between 0 and 1 to indicate how good is scene for starting, tracking or resetting scene perception.
1.0 -> represents ideal scene for starting scene perception.
0.0 -> represents unsuitable scene for starting scene perception.
a negative value to indicate potential reasons of a tracking failure.
-1.0 -> represents the scene lacks of structural/geometry information.
-2.0 -> represents The scene lacks enough depth data (too far away from or close to the camera.)
SampleProcessedEvent
a positive value between 0 and 1 to indicate how good is scene for starting, tracking or resetting scene perception.
1.0 -> represents ideal scene for starting scene perception.
0.0 -> represents unsuitable scene for starting scene perception.
a negative value to indicate potential reasons of a tracking failure.
-1.0 -> represents the scene lacks of structural/geometry information.
-2.0 -> represents The scene lacks enough depth data (too far away from or close to the camera.)
BlockMesh
ImageSize
Image
InitialConfiguration
false.
init is called, afterwards it will remains same throughout the runtime of scene perception module. The default value of voxel resolution is LOW_RESOLUTION.
[R | T] = [Rotation Matrix | Translation Vector]
where R = [ r11 r12 r13 ]
[ r21 r22 r23 ]
[ r31 r32 r33 ]
T = [ tx ty tz ]
[r11 r12 r13 tx r21 r22 r23 ty r31 r32 r33 tz]InterestRegion
MeshData
MeshingThresholds
MeshingUpdateInfo
countOfBlockMeshesRequired is set to true otherwise this value will be ignored. If set to true, on successfull call to this function it will update block meshes array.
countOfVerticesRequired is set to true otherwise the value is ignored. If set, on successful call to this function it will update vertices array.
countOfFacesRequired is set to true otherwise the value is ignored. If set, on successful call to this function it will update faces array.
MeshingUpdateConfigs
Normals
Point3D
Sample
SaveMeshInfo
SurfaceVoxelsData
numberOfSurfaceVoxels.
VolumePreviewData
Vertices
VoxelsDataConfig
getSurfaceVoxels function.
getSurfaceVoxels function.
CameraIntrinsics
Point2D
MeshingResolution
The high mesh resolution.
The median mesh resolution.
The low mesh resolution.
TrackingAccuracy
The high tracking accuracy.
The median tracking accuracy.
The low tracking accuracy.
The tracking is failed.
VoxelResolution
The high voxel resolution. Use this resolution in a object-sized scenario (1/256m).
The median voxel resolution. Use this resolution in a table-top-sized scenario (2/256m).
The low voxel resolution. Use this resolution in a room-sized scenario (4/256m).
PixelFormat
The 32-bit RGBA32 color format. When format of
an Image instance is set to rgba32,
the data of that image instance must follow the
bytes layout of Canvas Pixel ArrayBuffer
defined in [[!CANVAS-2D]].
The depth map data in 16-bit unsigned integer. The values indicate the distance from an object to the camera's XY plane or the Cartesian depth.The value precision is in millimeters.
// SP states and changing conditions.
// -- IDLE
// Before the pipeline is initialized.
// -- INITIALIZED
// Pipeline is initialized, SP module is paused.
// Possible event in this state:
// checking event(depth quality)
// -- STARTED
// SP module starts to work, receiving raw smaples and reconstructing the
// volume space according to them.
// Typically this state is triggerred when the depth quality
// is acceptable.
// SP module is on tracking, and on reconstructing if reconstruction
// flag enabled.
// Possible events in this state:
// sampleProcessed event(quality, tracking accuracy, camera pose)
// meshupdated event(no data)
// Interfaces get/queryXXX are used to get processed data from SP module.
// They are availiable after the module being initialized.
// Although, they can be accessed in both "INITIALIZED" and "TARTED" states,
// they may return unavaliable data if the SP module haven't successfully
// established the co-ordinate system from the first frame.
//
// getSample(processed sample including color/depth image)
// getMeshData
// get/query[XXX] interfaces to get configurations or data from SP module.
// The changing diagram.
// --sp.destory() from other states--> "IDEL" --sp.init()--> "INITIALIZED"
// "INITIALIZED" --sp.start()--> "STARTED"
// "STARTED" --sp.stop()--> "INITIALIZED"
// This example will show the skeleton frame to init and control SP module.
var sp = realsense.ScenePerception;
var initButton = document.getElementById('init');
var destroyButton = document.getElementById('destroy');
var startButton = document.getElementById('startSP');
var stopButton = document.getElementById('stopSP');
// Set the initial state of buttons.
resetButtonState(true);
function resetButtonState(beforeStart) {
initButton.disabled = !beforeStart;
destroyButton.disabled = beforeStart;
startButton.disabled = beforeStart;
stopButton.disabled = true;
}
initButton.onclick = function(e) {
// Please refer to InitialConfiguration for more info.
var initConfig = {
useOpenCVCoordinateSystem: true,
colorCaptureSize: {'width':320, 'height':240},
depthCaptureSize: {'width':320, 'height':240},
captureFramerate: 60
};
sp.init(initConfig).then(function() {
// Set the state of buttons.
resetButtonState(false);
// Other initialization work can be done here.
console.log('init succeeds');
}, function(e) {console.log(e);});
};
startButton.onclick = function(e) {
sp.start().then(function() {
startButton.disabled = true;
stopButton.disabled = false;
console.log('SP started successfully');
}, function(e) {console.log(e);});
};
stopButton.onclick = function(e) {
sp.stop().then(function() {
console.log('SP stops working.');
startButton.disabled = false;
stopButton.disabled = true;
}, function(e) {console.log(e);});
};
destroyButton.onclick = function(e) {
sp.destroy().then(function() {
console.log('stop succeeds');
resetButtonState(true);
}, function(e) {console.log(e);});
};
var qualityElement = document.getElementById('quality');
var accuracyElement = document.getElementById('accuracy');
sp.onchecking = function(e) {
var quality = e.data.quality;
qualityElement.innerHTML = 'Quality: ' + quality.toFixed(2);
};
// Please refer to SampleProcessedEvent inferface for event data.
sp.onsampleprocessed = function(e) {
accuracyElement.innerHTML = 'Accuracy: ' + e.data.accuracy;
qualityElement.innerHTML = 'Quality: ' + e.data.quality.toFixed(2);
sp.getSample().then(function(sample) {
// The sample object contains color image and depth image.
// The image data structure, which could possibly look like this:
// sample.color = {
// width: imageWidth,
// height: imageHeight,
// data: Uint8Array
// };
// sample.depth = {
// width: imageWidth,
// height: imageHeight,
// data: Uint16Array
// };
// Please refer to getSample interface for more details.
}, function(e) {console.log(e);});
sp.queryVolumePreview(e.data.cameraPose).then(function(volumePreview) {
// The volume preview image data structure is similar with color and depth image in sample.
// volumePreview = {
// width: imageWidth,
// height: imageHeight,
// data: Uint8Array
// };
// Please refer to VolumePreviewData interface for detail data format.
});
};
sp.onmeshupdated = function(e) {
thisObj = this;
sp.getMeshData().then(function(meshes) {
// In the meshes, there are information about vertices, faces and color.
// Meshes can be used to reconstruct the scanned scene by WebGL.
// Please refer to MeshData inferface for more details.
}, function(e) {console.log(e);});
};
sp.onerror = function(e) {console.log(e););