Feature Story

Seeing the Future: Why Graphics and Multimedia Have Barely Started Transforming Consumer Electronics

By Tony King-Smith, VP marketing, Imagination Technologies

It seems only yesterday…

… when phones started trying to be small PCs – a world where graphics processors (GPUs), video engines, sophisticated cameras and lots of processing power have been part of the PC's DNA for years. How could they possibly pack all that into a small hand-held, battery-powered device that fits in your pocket?

Funny thing about phones: consumers rarely test phones in the shop before they buy them for their #1 job: making phone calls. So as phones started to become high volume consumer products more than a decade ago, it was all about cosmetics – flip, clam, lozenge; black, pink or clip-case.

And then the world started to change. The Nokia N95 when launched in 2007 was one of the first true full-functionality smartphones, featuring not only all the functions we take for granted today like cameras, Wi-Fi and bluetooth, but also had an advanced colour screen with a simple but visually rich graphical user interface (UI) accelerated by a GPU.

Then came the iPhone. Instead of a complex menu-driven text-based UI, the whole user experience was transformed thanks to a highly interactive touch-screen based, highly graphical UI. This rich, intuitive user experience was used throughout every function and app the phone had, featuring visually stunning interactive natural movement-based visual effects. The iPhone was designed to turn the somewhat impenetrable functionality hidden in smartphones into something that users could immediately see and experience in store and use with effortless ease. That changed everything, and not just for phones.

The 3D graphics horsepower essential for enabling the high speed, ultra-smooth behaviour and visual richness of advanced UIs – such as the 'coverflow' paradigm used in many media players at that time - could also enable games, 3D navigation and more.

Once phone designers and app developers came to appreciate the potential of these mobile GPU-powered phones, they realised that these high performance graphics capabilities could have a massive impact not only on consumers' choice of product in store, but also enable substantial follow-on revenues from additional applications, UI upgrades and much more. And better still, they could now differentiate their products through their UIs, even though most of the phones were looking more and more physically similar, now they were dominated by a large high resolution touch screen – and not much else.

The lessons have been learned amazingly quickly, and now thanks to mobile GPUs and high performance mobile processors an ever widening range of CE products are coming to rely on graphics enhancing their usability and desirability.

What's in store for tomorrow?

The 'Pandora's box' of graphics has barely creaked open. From 2012 a new generation of graphics-centric devices will start to transform consumers' perceptions of what CE devices can do yet again.

New generation computing devices are eschewing typing for input based on touch-sensitive screens complemented by highly responsive interactive graphics. TV UIs are getting smart, connected and gestural. Tablets are starting to appear in more and more places and CE products of all kinds – not just phones and TVs but HiFi systems, remote controls, even white good such as washing machines and refrigerators are now sprouting screens sporting advanced touch UIs.

GPU power to the people

But beyond that, Graphics Processing Units (GPUs) that are powering all these amazing graphics UIs, games, navigation apps and more are about to also become key high performance processing powerhouses for tomorrow's CE devices. This is increasingly referred to as GPGPU (General Purpose GPU) processing power.

A GPGPU is just another name for a GPU that is programmed using a different API – it's exactly the same hardware. But now developers can make that same GPU work much harder, doing not only graphics but now an ever-expanding array of other more general purpose processing tasks.

Thanks to the explosion in high performance, ultra-low power GPUs such as Imagination's PowerVR GPUs now shipping in hundreds of millions of devices each year, new functions never thought possible in a mobile or consumer device are about to become commonplace. Augmented Reality (AR), gesture recognition (think Kinect on steroids), face and image recognition, real time image and video enhancement, physics simulation for more realistic photo-realistic games and apps – these are but a few of the myriad new applications now being developed feverishly in labs all over the world, as developers race to take full advantage of the enormous raw processing power now coming to consumer devices everywhere.

We've seen the demand graphics processors growing in the mobile space so fast that today it's not a question of whether or not a smartphone needs a GPU, but whether the GPU is powerful enough for the Apps consumers want to run. GPUs are now getting into lower end phones and 'just good enough' graphics simply isn't acceptable to consumers reared on iOS and Android.

Into the living room and the car

We are seeing the trends from mobile now spilling over into TVs and set-top-boxes (STBs) as people realize the user experience - driven by GPU performance - is becoming a fundamental feature in selling these products. As TVs start to employ more advanced visual UIs, their designers have come to see the same trend that is transforming the mobile industry: advanced UIs and great graphics controlling ever more complex functionality will make or break future TVs and STBs. The user experience is becoming the feature that defines the product, and will make people pay for one product over another.

What's more, just as touch-based UIs have transformed phones and computing devices by letting the user directly interact with visual information, the future TV screen will become an interface whereby gestures let users interact with on-screen information to control the TV and interact with applications. What was science fiction in the film Minority Report just a few years ago will be in products coming soon to a store near you.

GPUs are also finding their way into places you wouldn't expect them. In cars, for example, the rise of virtual dashboards and navigation is fast becoming an unstoppable trend, as car manufacturers come to appreciate the dramatic impact a graphics-driven multimedia-rich cockpit has on drivers and passengers alike. A multitude of in-car cameras will be feeding in masses of real-time visual information forming a 360 degree view of the surrounding environment into the cars' GPU-powered information and entertainment systems, to be used for everything from parking proximity to security and poor weather and night-time driver guidance. And despite all this new graphics horsepower these cars will actually be cheaper too, as customisation becomes more a matter of software updates than new boxes or switches, making global production and local customisation far more cost-effective than ever before.

Don't miss a 3D moment on your mobile

As consumer's appetite for S3D (stereoscopic 3D) content continues to rise, you'll be able to not only watch it all but capture and edit it too – all on your smartphone or tablet thanks to mobile GPUs. The ability to handle S3D is pretty much standard practice on GPUs now and new devices like the LG Optimus 3D don't even need glasses. So you'll be able to watch that latest 3D Hollywood blockbuster on the move as well as in your living room.

How fast is it?

Graphics has touched a nerve with consumers, and that means product reviewers everywhere want to say something about not only talk time and battery life, but how good the graphics is. Products reviews are starting to make reference to stats from benchmarking specialists such as Rightware and Kishonti, where they exercise the graphics capabilities of products that include the Khronos OpenGL ES graphics APIs using a series of specialised tests that are then compared in league tables.

Ray Tracing

Ray tracing is a technique for rendering three-dimensional graphics with complex and more natural lighting models to achieve cinema-quality 3D and a level of near photographic realism that is impractical with traditional 3D 'rasterisation' graphics techniques.

To date ray tracing has been used in specialized high end applications to create photo-realistic visual content – in usually far from real time. Computer animated movies such as Toy Story, mechanical and architectural CAD visualisation have traditionally used ray tracing techniques to create lifelike and photorealistic imagery. But it takes a long time and huge amounts of processing power to do so – rendering a single frame on a high production quality cinema film can take an hour or more on a massive rendering farm.

However technologies like Imagination's forthcoming Caustic ray tracing technology claim to radically lower the cost and dramatically increase the efficiency and performance of ray tracing. It's a few years away yet, but it's coming to the living room and ultimately to mobile – and when it does expect true photo-realistic interactive apps that deliver a whole new genre of visually rich interactive content.

Video

Video content delivery is the new TV and the 'network' delivering it will be increasingly the social one.

Tablets and phones are about to deliver full HD1080p video so efficiently (thanks to ultra-low power dedicated on-chip video processors, sometimes known as VPUs) that you can watch, manipulate and browse high quality videos to your hearts content and it won't kill the battery. And with advanced UIs you can browse all your video collection - with simultaneous live previews of all of them just as you browse photos today.

High-quality multi-stream and multi-standard VPUs are now making it possible to play videos from any source on any device, whether mobile, in the living room or in-car – vital in this age of un-tethered user content.

The 3D Web

Rich graphics is not just limited to specialist apps such as games and UIs. All that visual 'wow factor' content is about to appear in your browser too.

Thanks to HTML5 adopting APIs such as WebGL, developers will now be able to make far more advanced graphical content work in browsers across a broad range of platforms and OSs. And it's happening in Android too, with support for the Khronos OpenGL ES 2 API fully optimized throughout the OS.

For standard-alone apps, Adobe's latest Flash 11 release, available using their AIR3 technology on a broad array of mobile as well as desktop platforms, delivers graphics acceleration thanks to their Stage3D 'Molehill' extensions to ActionScript 3. It has already been demonstrated by gaming greats such as Epic running Unreal Engine 3.

With the popularity already of social network based games the rise of 3D on the web will lead to an exciting new level of games, more in line with what is possible on today's games consoles and smartphones.

And of course as browsers break free of computers and inhabit more and more CE products, they are bringing the 3D web with them.

GPGPU – parallel processing for all

The GPU is transforming how programmers write applications. Tomorrow's apps will need not only amazing graphics, but they'll also need to perform massive amounts of local real-time processing. Fortunately, just as mobile CPUs have matured to the point they are approaching their desktop cousins in sophistication and performance, mobile GPUs are starting to deliver unheard-of floating point processing power (often measured in GFLOPS – Floating Point Operations per Second). But why are GPUS delivering so many more GFLOPS than CPUs?

The reason for this is simple: a CPU is essentially a sequential processor. Even dual core or quad core CPUs still execute a series of sequential tasks – adding more CPU cores just enables it to execute a few more of these tasks at the same time. So multi-core performance of CPUs does not scale linearly with a multicore CPUs – a typical quad core CPU is likely to deliver perhaps 2.6x a single CPU core, except for highly specialised applications such as data servers or routers.

A GPU on the other hand is a true parallel processor. Each 'execution pipe' of a GPU executes a different instruction at the same time as its neighbour. However, since GPUs are designed to work on lots of data in parallel at the same time (e.g. different pixels on a screen), the more GPU pipes you have, the more performance you get – it pretty much scales linearly if you have the memory subsystem to feed it. So the more pipes you can get on a chip, the more powerful the GPU. And with silicon processes still getting more dense every year, GPUs are the ideal way to take maximum advantage of these continuously increasing levels of semiconductor integration.

Over the next few years the raw processing power of GPUs will come to dwarf CPUs by at least 10x-100x. Until now the specialized nature of parallel processors and the languages used to program them has meant their use has been restricted to low volume niche applications like weather prediction or scientific simulations. However, since there are now hundreds of millions of mobile and CE GPUs being shipped every year, and new industry standard programming APIs such as OpenCL and Renderscript Compute available to code with, we're set to see yet another software revolution – the era of ubiquitous parallel processing.

A GPU-driven future

Demand for graphics in mobile and CE products started from a simple but universal desire from consumers for more usable technology that's fun, intuitive and pretty. Touchscreens displaying enticing, dynamic content will always attract people's attention to products, just as good industrial design or new easy-to-understand features always have.

Now that GPUs and graphics are at the heart of tomorrow's consumer experiences and driving their purchasing decisions, a whole new GPU-powered world is opening up. It promises to be a beautiful journey!

Read this article interactively

Return to: 2012 Feature Stories