Fast and Efficient Online 3D Reconstruction on Resource-Constrained Devices using Implicit Neural Representations

3D content creation (scenes and objects)

As part of one of the world's most innovative and recognizable brands, Sony is committed to support university research and innovation in the U.S., Canada, India, and select European countries, while also fostering partnerships with university faculty and researchers. The Sony Research Award Program provides funding for cutting-edge academic research and helps build a collaborative relationship between faculty and Sony researchers. This Sony Research Award Program research collaboration between Sony and University of Toronto is led by Masaru Ikeda of Sony and Professor Nandita Vijaykumar of University of Toronto.

Masaru Ikeda: Can you provide a brief description of your research collaboration project, specifically speaking about differentiation from the state of the art?

Professor Nandita Vijaykumar: Implicit neural and data-driven representations of 3D scenes and objects (e.g., neural radiance fields and 3D Gaussian Splatting) have emerged as a transformative approach for 3D reconstruction and novel view synthesis. Neural representations are AI-based methods that learn to represent the shape and appearance of objects and scenes as compact mathematical models, allowing computers to understand and recreate them in 3D with high precision. They have gained importance because they enable highly detailed and efficient 3D reconstructions, simulations, and visualizations, which are essential for applications like virtual reality, robotics, autonomous navigation, and medical imaging. These representations make it possible to efficiently render, manipulate, or analyze 3D content in ways that traditional methods cannot. Two important challenges with enabling these representations are: First, training these representations is slow and compute-intensive. Second, these representations are challenging to edit. This research collaboration aimed at addressing both challenges, and to this end, we proposed three novel techniques to enable efficient and editable implicit representations. At the time this research was conducted, ours was the first work to profile and accelerate 3D Gaussian Splatting training and to enable distribution of computation between cloud and edge compute. The completed research was published in the following top-tier conferences and journals: Neurips 2024, ASPLOS 2025, and RA-L 2025.

Masaru Ikeda: How would the technology developed from your research collaboration impact the future of content creation, and can you describe the various potential use cases for your technology?

Professor Nandita Vijaykumar: The technologies developed in this research collaboration have huge potential to impact the future of content creation. Two of the main research directions involved accelerating the creation of the 3D content (i.e., during the training process). The third direction involved enabling editing which is a key requirement for content creation. The proposed technologies can be used for high-speed 3D scene reconstruction, live 3D streaming, and interactive object selection and segmentation from a scene.

Masaru Ikeda: Could you please describe the impact of close collaboration with Sony scientists on your research and research direction?

Professor Nandita Vijaykumar: The close collaborations with the Sony researchers was helpful in determining the usefulness and impact of different research directions, enabling us to pursue the most impactful research problems. The Sony researchers also provided valuable feedback on the code implementations, potential challenges in implementing new solutions in real products, and pointers to related projects and papers. The meetings with the Sony researchers were also valuable learning opportunities for the students involved in the project and were very helpful in brainstorming solutions and next steps.

Masaru Ikeda: In your opinion, can you describe the obstacles to get this technology to mass adoption?

Professor Nandita Vijaykumar: One of the proposed ideas (efficient atomic reduction for Gaussian Splatting) was already incorporated into the popular open-source infrastructure, Gsplat. We believe the technologies we developed have the potential for immediate impact and we have made code available to this end. One of the challenges in getting any new technology to be used widely is generating enough visibility about the technology’s benefits. Hence, collaborations with leading industrial partners, such as Sony, are great opportunities for tech-transfer to turn research ideas into real world solutions.

Masaru Ikeda: Can you describe the impact you expect that the Canadian Government’s NSERC Grant matching funds, obtained with Sony’s support, has had/will have on your research goals and outcome?

Professor Nandita Vijaykumar: Canada’s NSERC Alliance program encourages university researchers to collaborate with industry partners by double matching any funding provided by the industry partner. We were successful in having this grant approved for our Sony collaboration. This grant was instrumental in widening the scope of the project, by enabling us to support more graduate and undergraduate students and purchase the necessary compute infrastructure.

Sony's research team on this project