Summary
- Sony patented a multi-modal solution for adding virtual buttons to physical controllers.
- The patent describes a wide variety of finger tracking implementations, ranging from superficial Bluetooth signal analysis to AI-powered systems.
- The IP filing underlines Sony's continued interest in finger tracking, a technology it has yet to commercialize even after years' worth of R&D efforts.
Sony is possibly developing a solution for equipping physical controllers with virtual buttons using a variety of technologies and combinations thereof. The hardware-agnostic concept offers some new insights into Sony's ongoing interest in finger tracking solutions.
The PlayStation maker has been experimenting with finger movement detection techniques for years. A recent uptick in Sony's patent activity even suggested that the PS VR2 might support finger tracking, but that didn't come to pass. As of late 2023, the company has yet to commercialize any such solution.
Sony Patents Spherical Controller
Sony files a pair of brand new patents for a spherical PlayStation 5 controller that could add another level of immersion to VR gaming.
And yet its research and development efforts on this front continue to this date, as suggested by a newly secured patent describing a way to extend physical controller capabilities using hand and finger tracking in order to achieve virtual button functionality. Specifically, Sony patented a multi-modal approach to tracking virtual button presses that can leverage anything from wireless receivers and microphones to smartphone cameras and inertial measurement units, devices that track body properties such as force and angular rate of rotation.
Sony's Patented Virtual Button Tracking Methods
- Analyzing a camera video feed from a phone, console, or webcam.
- Analyzing distance sensor data.
- Analyzing inertial measurement unit (IMU) data.
- Analyzing pressure sensor data.
- Analyzing sound data.
- Analyzing Wi-Fi signals, Bluetooth data, or both.
Several implementations described in the newly emerged patent incorporate an analysis of channel state information (CSI) in order to detect both large and subtle gestures. By tracking amplitude variations of Wi-Fi or Bluetooth data packets exchanged between a transmitter and receiver, which can be anything from a controller to a smartphone, Sony believes it can accurately detect actions like waving, typing on a keyboard, and pressing a specific spot on a gamepad, all of which have distinct CSI footprints.
True to Sony's long-standing interest in machine learning, the freshly surfaced patent describes a number of implementations that would leverage an AI model in order to intelligently interpret sensor data. However, the company believes that even a rudimentary analysis of things like Wi-Fi signals and camera feeds could be used for achieving virtual button functionality. One such solution describes a controller-mounted smartphone that simply records the user's finger movements, then translates them to button presses.
Sony's Virtual Button Tech Is Super Broad, Even By Patent Standards
The overall solution is extremely broad, even by patent standards. The generic nature of Sony's latest patent suggests that its underlying technology is unlikely to ever be commercialized in its entirety, as some of its described implementations are bound to be more practically viable than others. Nevertheless, the sole existence of this IP filing indicates that Sony is still very much exploring the idea of adding finger tracking to its electronics. Time will tell whether this ongoing R&D effort yields a new version of the DualSense or a completely novel category of gaming accessories.