We love to see what our users make with ToF AR.
Earlier this year, Sony Semiconductor Solutions published a profile of two of our R&D colleagues who work on innovative ways to combine imaging and other sensing technologies. One of the engineers interviewed was Eigen Yusaku, who has been involved in developing and improving the ToF AR SDK, but who also develops his own apps using ToF AR.
Eigen recently released Pistol Bang Bang for iPhone, and we took the chance to find out more about Eigen, why he enjoys creating mini games with ToF AR… and what’s next.
Tell us about Pistol Bang Bang. What are the main features of your app?
Pistol Bang Bang is a mini game that uses hand tracking. There are actually two games that let you use your hands as controllers. You can either pretend to shoot a finger gun or play rock-paper-scissors with a twist. The twist is that you have to lose to win! I made that one a “reverse win” to make it more like a brain training app. It sounds easy, but it’s harder to think in reverse logic.
Who are the main target users for Pistol Bang Bang?
Anybody! Users of all ages! Like I said, I had brain training in mind as well, but anybody can play. It’s fun to play with friends, too. There aren’t too many downloads, but I created it as a trial app, and I also wrote a blog post on how I made the app (in Japanese), which already has more than 300 views.
What makes you want to create apps with ToF AR?
At first, there was little interest for the ToF AR SDK, even in Sony, but I wished more people would give it a try. I was interested in an SDK with depth-sensing capability and wanted to use tracking like motion capture and motion tracking.
You can do motion capture with ToF AR?
Absolutely. ToF AR helps a developer to create all kinds of apps easily. There are a lot of components already available. For example, ToF AR can recognize 13 or 14 different hand gestures like bending fingers, OK, rock, paper, scissors, and can track hand movements like snapping your fingers.
What other AR/VR SDKs have you tried?
There are not a lot of SDKs with depth capture capability. I’ve used AR Kit, AR Foundation, and others. You can do depth capture with LiDAR, but ToF AR can use it for 3D tracking with high precision, which makes it quite unique.
Tell us about your experience as a developer. What did you do before you joined Sony?
I used to make apps for smartphones. As a student, I spent about a year creating and publishing Unity apps with friends and, since joining Sony four years ago, I’ve been creating more Unity apps, so I have about five years of experience with Unity.
What advice would you give to developers considering developing apps with ToF AR?
My advice would be not to focus on depth sensing, per se, but consider using ToF AR for motion capture of the face, body, and skeleton. It’s very useful for VTubers, too. ToF AR is fantastic for high-precision tracking and motion capture.
What skills, like specific programming knowledge, are required for AR app development?
Don’t worry, it’s not hard to get started. All you need to do is place out some prefabs where you want them, and you’ll get the data you need for your app. You don’t need any specific programming knowledge.
Did you use ToF AR Server to debug? What was your experience with that?
Yes, I did. I think ToF AR Server makes a big difference in efficient development. ToF AR needs the smartphone sensor data, and you can’t get that from a simulator. It’s really annoying to build and install your app every time you want to test it. But with ToF AR Server, you can just connect your phone and PC using Wi-Fi or USB and run the app in your editor with the real sensor data from your smartphone. It saves you a lot of time and hassle.
Is there something you wished ToF AR had or was different?
Above all, I wish there were more tutorials to show how to use the components to create different kinds of apps. Right now, we only have Depth tutorials, but I would like to see tutorials on how to use depth to scan a room, or something else that can be used for real app development. We do provide sample apps, but then developers need to go through the source code by themselves.
Did you come across any issues or challenges when you were developing Pistol Bang Bang?
Not really. ToF AR is pretty easy to use, and I had fun making the app. It wasn’t a big deal at all.
Pistol Bang Bang is currently only available for iOS. Do you have plans to release a version for Android?
The reason I created it for iOS is that right now a good quality front camera with depth information is available almost exclusively on Apple devices. Only a few Android devices have similar cameras. So, any plan to release it for Android depends on hardware being available.
Do you have any plans to develop more apps made with ToF AR?
Actually, yes! It’s in development now, and might even be ready to release this month. Stay tuned!