Enhancements to the Three.js 3D Tiles Renderer
3DTilesRendererJS is a library originally developed at NASA/Caltech-JPL for rendering 3D Tiles in Three.js. The project was released as open source and is now maintained by Garrett Johnson, who worked there for more than a decade.
In April 2024 Garrett Johnson was awarded a Cesium Ecosystem Grant to expand support in 3DTilesRendererJS for new 3D Tiles metadata extensions and to enable data overlays useful for understanding 3D Tiles Mars terrain data from NASA JPL. Those improvements have now been implemented; this post by Garrett describes the work completed.
This year I was grateful to have been awarded a Cesium Ecosystem Grant to enhance 3DTilesRendererJS, an open source library originally developed by NASA/Caltech-JPL (Jet Propulsion Laboratory) for rendering JPL-generated Martian datasets. The project enables efficient rendering and interaction of 3D Tiles in Three.js, an open-source, cross-browser JavaScript library for 3D graphics in a browser. Thanks to the work supported by the grant, the library now supports glTF metadata extensions from 3D Tiles 1.1 and alternate tile texture overlays. These new features will enable end users to implement more robust AEC and GIS applications with 3D Tiles and Three.js.
3D Tiles Metadata Additions
3DTilesRendererJS now supports the glTF EXT_structural_metadata and EXT_mesh_features extensions. The mesh features extension allows for parsing and reading associated IDs with varying levels of granularity in a model, which can subsequently be used for looking up more complete data. And the structural metadata extension gives users the ability to look up detailed, structured data based on embedded identifiers, including those from mesh features.
Visualization of structural metadata and mesh features from 3D Tiles 1.1 is now possible in Three.js with the San Francisco Ferry Building.
The plugins for the EXT_structural_metadata and EXT_mesh_features extensions compatible with Three.js’ GLTFLoader are located in the src/three/loaders/gltf project directory, along with associated type definitions exported from the package. To use the plugins they must be registered to a GLTFLoader instance used for loading the files after creating the TilesRenderer instance for loading a tileset:
import {
TilesRenderer,
GLTFMeshFeaturesExtension,
GLTFStructuralMetadataExtension,
} from '3d-tiles-renderer';
// ...
// Create a tiles renderer and intialize a GLTFLoader with both
// metadata extensions
const tiles = new TilesRenderer( url );
const loader = new GLTFLoader( tiles.manager );
loader.register( () => new GLTFMeshFeaturesExtension() );
loader.register( () => new GLTFStructuralMetadataExtension() );
tiles.manager.addHandler( /(gltf|glb)$/g, loader );
Once the tiles are loaded the data from the extensions can be accessed via the meshFeatures and structuralMetadata fields in the associated objects' userData. These fields can be accessed when hovering over or clicking on a tile after a raycast, for example, to display metadata tables or highlight the object by id:
const triangle = new THREE.Triangle();
const barycoord = new THREE.Vector3();
const invertedMatrix = new THREE.Matrix4();
domElement.addEventListener( e => {
// ... set up raycaster
const hits = raycaster.intersectObject( scene );
if ( hits.length > 0 ) {
const { object, face, point, faceIndex } = hits[ 0 ];
// ... calculate the barycoord of the point on the geometry face
const { meshFeatures, structuralMetadata } = object.userData;
if ( meshFeatures && structuralMetadata ) {
// get the metadata features and property table indices
const features = meshFeatures
.getFeaturesAsync( faceIndex, barycoord );
const tableIndices = meshFeatures
.getFeatureInfo().map( info => info.propertyTable );
// read and log the structural metadata
const data = structuralMetadata
.getPropertyTableData( tableIndices, features );
console.log( data );
}
}
} );
Metadata Demo & Documentation
Two 3D Tiles datasets from Cesium ion are available for demonstration here; to use the demo please retrieve the trial ion access token from the Cesium repository. Refer to the example implementation for code showing how to use custom materials for tileset brushing and reading data, and see supporting documentation for API documentation and function definitions.
Additional Three.js Contributions
To support the implementation of the new extensions, a variety of contributions were made to the broader Three.js ecosystem to enable improved performance for reading data from textures and to add new math utilities. Changes were made to allow copying texture subframe data to the GPU to avoid stalls from uploading a whole texture. This allows for only uploading and reading a single pixel of data so it can be quickly read back when querying mesh features and structural metadata textures without blocking page interactivity.
Likewise, thanks to initial work from @sciecode on Github, GPU texture data buffers can now be read back to the CPU asynchronously in Three.js. This is used for reading texture-encoded metadata values without waiting for remaining GPU work to complete, allowing for a significantly smoother user experience when reading the values.
And lastly, a new Matrix2 core math data type has been added to the library. Matrix2 is a type supported by glTF and WebGL vertex attributes, in addition to being used to store metadata values and is now available to be used and expanded on in Three.js.
Texture Overlays
At NASA JPL, texture overlays are used on top of terrain geometry to help provide more context for robotics and science operations. The texture overlay plugin is designed to support loading one or more replacement textures for a set of tiles to display alternate data overlays–including properties like slope, roughness, or error metrics. This feature will extend to other pre-processed texture data visualizations for photogrammetric tilesets should they provide alternate textures per-tile. By providing the ability to replace textures for the full tileset rather than using a 2D projection, overlay detail can be included across the full extent of the tileset, including concave geometry surfaces. For this demonstration a Martian tileset was produced using NASA JPL tools and data from the Mars 2020 mission using slope magnitude as an example overlay. In practice NASA JPL uses a variety of these kinds of precomputed texture skins to aid with operations on Mars, including displays for robotic kinematic reachability or surface roughness.
Check out the demo for the data overlay, which shows loading ten wedge tilesets with slope magnitude overlay textures. One or more of these alternate textures can be blended with the original using custom shaders or used as opaque replacement textures for the tiles.
Texture overlay of Mars data showing multiple sets of overlays including slope magnitude. Left: original textures, center: blended slope visualization, right: opaque slope texture overlay.
To produce a sample dataset for this feature and ensure functionality for the practical operational use case, the open source NASA-AMMOS/Landform project, used internally by JPL to generate merged and per-wedge 3D Tiles datasets for Mars 2020 missions operations, was used to generate the sample terrain. Color imagery and alternate image products from the official NASA JPL Planetary Data Systems (PDS) data archive were used, which hosts historical data from a variety of NASA missions for download. For the sake of demonstration only a single alternate set of textures highlighting slope magnitude have been produced, though other alternate textures can be generated from available PDS data.
Documentation for the TilesOverlayPlugin is available on the plugins documentation page. If your dataset contains alternate sets of textures, the plugin can be instantiated and textures applied to tiles materials like so:
import { Tiles Renderer } from '3d-tiles-renderer';
import { TextureOverlayPlugin } from './examples/plugins/TextureOverlayPlugin.js';
// ...
const tiles = new TilesRenderer ( url );
const plugin = new TextureOverlayPlugin( {
textureUpdateCallback: ( scene, tile, plugin ) => {
// Retrieve the loaded texture layers for this tile
const slopeTexture = plugin.getTexturesForTile( tile ).slope;
scene.traverse( child => {
if ( child.isMesh ) {
// Replace the original texture with the new one
child.material.map = textures.slope;
}
} );
},
} );
// Set up a new layer textures to trigger when a new tile geometry is loaded
plugin.registerLayer( 'slope', url => {
// Adjust the tile geometry url to load the associated texture
const url = tileUrl
.replace( '/tilesets/', '/textures/slope/' )
.replace( /\.glb$/i, '.png' );
// Load the texture and fix the texture settings
return new TextureLoader()
.loadAsync( url )
.then( tex => {
tex.colorSpace SRGBColorSpace;
tex.flipY = false;
return tex;
} );
} );
tiles.registerPlugin( plugin );
To support this work, a new extensible plugin system was added to the TilesRenderer class, which has subsequently been used for the development of other new plug-and-play features for 3D Tiles.
Additional Plugins
The addition of a new TilesRenderer plugin system for tile overlays has enabled a new addon architecture for 3D Tiles rendering. Previously a full subclass of the TilesRenderer class would need to be made, making it very difficult and brittle to use multiple extended classes at once.
The new TilesFadePlugin has been added to smooth the transition between tile levels of detail by modifying materials to add dither animation when fading in and out. The TilesCompressionPlugin compresses loaded tile geometry data and reduces texture mipmap usage, which can improve GPU memory usage by over 30% in some cases. And the UpdateOnChangePlugin can improve performance per frame by only triggering tileset traversal when the camera or tileset has changed.
These plugins and others are available in the examples and core directory of the project. In the future, plugins could be added for other features including Three.js BatchedMesh support, tile geometry modification, new 3D Tiles features, and other user plugins.
Conclusion
With the work enabled by this grant the NASA-AMMOS/3DTilesRendererJS project now supports a variety of new features expanding the utility for scientific as well as architectural use cases. The tiles renderer project has benefited greatly from reports and contributions from the community to improve it over time so please feel free to provide feedback on the new features and try it out and let us know what you think.
Learn more about the Cesium Ecosystem Grants program and other ways Cesium engages with the broader community to see how you can get involved.