From Smartphones to AR Headsets, Xiaomi Invests in French Startups Betting on Sensor Revolution

This is Luca Verre‘s second visit to China this year. He came to attend the World Mobile Communications Congress held in Shanghai, China. In an interview with Jiemian News, he mentioned that besides MWC, his itinerary for this trip to China also includes following up on projects with the Shanghai team and exchanging ideas with investors.

Luca Verre is the CEO of sensor company Prophesee and also a co-founder of this Paris-based startup. Among numerous semiconductor startups, Prophesee’s venture into vision sensors remains a relatively niche track that is less known to the outside world. However, with its unique technology called “event-based sensors,” it has the potential to create new application fields.

The so-called event-based sensors (EVS) originated from the signal simulation of visual neurons and are a biomimetic sensor technology that imitates the light-sensitive mechanism of the eye. When the human eye senses light, the receptor cells on the retina convert the light signals into neural information. The subsequent neural cells selectively pick out bright and dark areas and transmit this information to the brain’s visual cortex through ganglion cells.

EVS is like the human visual nerve, where the light entering the sensor is converted into electrical signals. The electrical signals are separated by a comparator based on brightness changes, forming bright and dark signals. Then, through subsequent signal processing, they are outputted in the form of EVS image data. Therefore, EVS is a type of visual sensor that is only sensitive to brightness changes and outputs images.

SEE ALSO: Xiaomi Car Announces Multiple Senior Executive Adjustments

Luca Verre introduced that the advantage of event-based vision technology lies in capturing dynamic information and temporal information. These pieces of information can complement well with the information from CMOS RGB sensors, or complement with 3D information collected by ToF or LiDAR. The combination of these sensor modes can form more valuable solutions.

The manufacturing of EVS, it was mainly completed with the help of Sony. The produced EVS sensor products are provided to customers. Currently, on the official website of Sony’s semiconductor division, there are already two models of EVS sensors available for external use. According to Prophesee’s introduction, leveraging Sony’s experience in producing image sensors enables large-scale production of EVS sensors, achieving excellent shooting performance and cost reduction.

In fact, as a new type of sensor, the most important thing for EVS is to find practical applications. According to research and development personnel from Sony’s semiconductor division, the biggest challenge was that after introducing EVS to customers, everyone initially showed a strong interest in this unique sensor and found it fascinating. However, when it came to discussing its applications, customers often didn’t know how to use it effectively, making it difficult to move on to the next stage.

This is also a question that Luca Verre often needs to ponder.

The sensor subdivision is a specialized and complex field, seemingly distant from everyday life. However, some of its applications are closely related to the lives of ordinary people. For example, in devices such as smartphones and cars, the embedded image sensors serve as the “eyes” of these devices. In the past, traditional CMOS sensors (CIS) were mainly used. According to market research company Counterpoint’s data, the global shipment market size for CIS is projected to reach 19 billion USD by 2022. Among them, mobile phones account for the highest share at 13.2 billion USD.

Installing a different “eye” on smartphones is the achievement that Prophesee aims to reach in the consumer market. They have chosen the commonly used function of smartphone cameras, “deblur,” as their starting point. The basic principle involves combining data from EVS sensors with traditional sensors, allowing the system to fill in the gaps between two frames with events recorded by EVS in microseconds. Through algorithms, pure dynamic information can be extracted and motion blur can be repaired. Based on the demo provided by Prophesee to Interface News, this feature is quite impressive.

SEE ALSO: Xiaomi Reports Q1 2023 Performance: Progress in Chip Production, Electric Vehicles, and AI

The collaboration with Qualcomm will also lower the difficulty for EVS to enter the mobile phone market. “We are working with Qualcomm and some other smartphone OEM manufacturers to combine our sensors with conventional RGB sensors, in order to enhance the image quality of mobile photography and reduce image blurring,” said Luca Verre.

After integrating EVS sensors and Prophesee’s deblurring algorithm with Qualcomm’s Snapdragon mobile platform, it is advantageous to further open up the market. “Our cooperation with Qualcomm is of great strategic significance. Qualcomm is a market leader in high-end flagship smartphone application processors. By combining Prophesee Metavision sensors with Qualcomm Snapdragon SoC, we can improve the image quality of smartphones and promote our solutions to some smartphone OEM manufacturers,” said Luca Verre, emphasizing the long-term collaboration with Qualcomm and hinting that EVS-enabled phones will be launched soon.

In addition to smartphones, VR/AR headsets have also been a product of close attention for Luca Verre. In the early morning of June 6th Beijing time, Apple launched its first XR (mixed reality) wearable display device, Apple Vision Pro, and operating system VisionOS. It is set to debut in the US market earlier in 2024 and then expand to other countries. Vision Pro combines internal and external ecosystems and incorporates new interactive methods such as virtual keyboards, eye tracking, and gesture control to be applied in office work, entertainment, and other scenarios. Undoubtedly, this will also require a considerable demand for sensors.

The Apple Vision Pro headset has sparked widespread attention to XR topics, especially with its excellent performance in eye tracking and gesture recognition. This is considered a key interaction technology for the so-called ‘spatial computing’.

Both smartphones and XR are part of Prophesee’s layout in the consumer market. “Many XR manufacturers have shown strong interest in our event-based visual technology, and currently we are actively discussing and evaluating relevant application scenarios with some customers,” said Luca Verre. The curiosity towards XR products brought about by Apple’s demonstration effect has also served as a reference for newcomers. Based on discussions between Prophesee and XR manufacturers, eye tracking, hand tracking or gesture control, as well as real-time positioning and map building (SLAM), will be the applications where EVS sensors can be utilized in the field of XR.

In terms of product and capital relations, Prophesee has close connections with China. In order to further enhance its competitiveness in the relevant markets, Prophesee completed a Series C financing round of 50 million euros in July 2021. The investors include Sinovation Ventures from China, Huiyou Capital, Xiaomi, and Shanghai Inno-Chip.

Luca Verre said that China is one of the most promising markets for Prophecy. Potential target industries include smartphones, XR (extended reality), IoT (Internet of Things), robotics, and electric vehicles, all of which demonstrate enormous market potential. It is expected that these markets will continue to develop rapidly in the future. According to him, Prophesee has already established an office in Shanghai, enabling the company to have local team support in China. Additionally, he plans to open an office in Hong Kong.

Besides that, we are also in contact with some Chinese investors and preparing for a new round of financing,” said Luca Verre.