New 'Furion' software allows untethered, high-quality VR

The virtual reality industry is faced with a conundrum: Users must be tethered to a server or PC in order to play high-quality apps.

At the same time, next-generation smartphones and wireless networks will not be advanced enough to sever the tether, say researchers who propose a new software platform to solve the problem.

"We have performed a systematic design study of the 'elephant in the room' facing the VR industry: Is it feasible to enable high-quality VR apps on untethered mobile devices such as smartphones?," said Y. Charlie Hu, a Purdue University professor of electrical and computer engineering. "Today's mobile hardware and wireless networks are about 10 times too slow for high-quality, immersive VR."

Yet, waiting for future mobile hardware or next-generation wireless networks is unlikely to help because of power limitations and greater computational demands needed for processing packets under higher data rates, he said.

Instead of relying on hardware improvements, researchers are proposing a three-step software solution. The platform, called Furion, allows for untethered playing of high-quality VR games using a smartphone.

Findings showed the platform achieves the "quality of experience" needed for VR apps on today's mobile hardware and wireless networks. Quality of experience, or QoE, is a measure of user satisfaction.

The research was detailed in a paper presented in October during the ACM MobiCom 2017 conference. The paper's lead author was Zeqi Lai, a graduate student at Tsinghua University who performed the research while a student at Purdue. A complete list of co-authors is included with the abstract.

The research team tested Furion with popular high-quality VR games Viking Village, Corridor and Nature.

For the QoE to be acceptable, each VR frame must be rendered at a rate of 16 milliseconds, or 60 frames per second. However, trying to render at this speed quickly exhausts the capacity of a smartphone's central processing unit; Google's Pixel XL is only capable of a speed of 111 milliseconds per frame.

Today's high-quality VR systems consist of a headset and server, which contains a powerful graphical processing unit, and the user is tethered to the server. One strategy to allow for untethered operation might be to render all of the frames on the server and transmit the frames over WiFi to the smartphone. But this takes even longer: around 200 milliseconds per frame at the highest data rate of WiFi supported by current smartphones.

"A key observation we made is that waiting for next-generation wireless networks such as 5G will not help because packet processing at 10 times higher data rate will exhaust the CPU on today's smartphones," Hu said.

Meanwhile, stagnating lithium-ion battery technology will limit next-generation smartphone hardware performance.

"Battery capacity in mobile devices has barely doubled over the past 15 years, and this has limited the CPU of smartphones from getting faster," Hu said.

At the same time, the clock rate of GPUs, which is critical to graphics performance, also has not improved much in recent years.

One reason for the heavy computational workload of VR apps is the constant need to render updates to the background environment in the virtual world. However, the background environment is largely unchanged from frame-to-frame – mountains and landscape, for example, remain much the same – and this background changes primarily in relation to the user's position.

"But the user's position doesn't change randomly," Hu said. "You move continuously and in a very predictable way. So that means we can predict how the background will change based on the user's position and pre-render the background."

Furion splits up the rendering, performing the background rendering on the PC or server and the less computationally heavy rendering of the foreground in the smartphone or other mobile device. This "cooperative rendering" approach – or pre-rendering the background on the PC and rendering the foreground on the smartphone - speeds the frame-rendering time to 14 milliseconds on Pixel XL and satisfies the QoE of high-quality VR

The entire background is rendered as a panoramic photo and split into four images, each decoded on one of the smartphone's four microprocessor cores. Then, the background can be automatically cropped to match the users changing viewing angle.

"You may suddenly turn your head, so if we render the whole panoramic frame for a location in the virtual world, it can simply be cropped properly to match wherever you are looking," Hu said.

In today's VR systems, the frames are handled in a five-step process beginning with the smartphone "fetching" a rendered frame from the server. However, Furion "pre-fetches" the background by anticipating and performing these fetch commands ahead of time. This approach allows the smartphone to keep up with rendering high-quality VR content.

"Prefetch means you predict the fetch and you start asking the server to get it before the VR game logic actually asks for it," Hu said.