Reflections by Susie Wee

A technical thread that I’ve been working on for a number of years is media security for scalable media. As part of this I have been developing a media security standard for JPEG-2000 images called JPSEC. This is an international team effort as JPSEC was developed by technologists from around the world.

This week I’ve been attending the JPEG/MPEG international standardization meeting in San Jose, California. It’s been a long process, and at this meeting we just received the notice of publication for JPSEC!!! This means that the standard is finalized and you will soon be able to get the JPSEC specification from the ISO the same way that you get the JPEG and MPEG coding specifications.

In this post, I provide an introduction to JPSEC with some background and motivation. I’ll dive into more details in future posts.

JPSEC security tools

JPSEC is Part 8 of the JPEG 2000 family of standards. JPSEC specifies ways of applying security to JPEG-2000 coded images. The three types of security tools specified included in the normative part of the standard are: confidentiality, authentication, and integrity. While these are very standard security tools that are commonly used in many applications, the thing that is different in JPSEC is how these tools are applied to media. Specifically, JPSEC applies security tools to JPEG-2000 images in a media-aware way.

Media-aware security

The traditional way to apply security to media would be treat the media data like any other data file and secure the entire file in a media-unaware way. If a security tool such as encryption is applied in a media-unaware way, then any structure in the media data would be lost.

However, some structure in the media data can be quite useful. For example, scalable coding methods code images into a bitstream that has a structure that makes it easy to access to a low-resolution version of the image, without requiring one to decode or transcode the entire bitstream. If this image data is encrypted for confidentiality in a media-unaware way, then the ability to extract the low-resolution version would be lost, or it would require you to decrypt and then extract the low-resolution data. However, once the image is decrypted, it is no longer secure.

On the other hand, if media-aware security tools are used, then security can be applied in a way that preserves the useful structure in the media. JPSEC recognizes the fact that media data actually has some useful structure to it, so it specifies how to apply security tools to JPEG-2000 images in a media-aware way.

Scalable coding of media: The structure of JPEG-2000 image data

JPEG-2000 has a particularly useful structure because it was designed to be “scalable”. While people often talk about the compression performance of JPEG-2000 vs. JPEG and other image coding methods. I think one of the biggest advantages of JPEG-2000 is its built-in scalability.

Scalable coding methods code media (images, video, or audio) in a manner that makes it easy to extract and decode different versions of it. For example, a scalably coded image can easily be decoded in high or low resolution. Decoding the image in high resolution involves decoding the entire bitstream. Decoding the image in low resolution simply involves extracting and decoding the low-resolution segments of the coded media data.

JPEG-2000 is a scalable image coding method. JPEG-2000 was designed in a way that makes it very easy to extract and decode a resolution, a tile, a color component, or a quality layer of the coded image. This can be done by simply scanning the bitstream, identifying and extracting the desired segments of the bitstream, and decoding those segments. This ability of transcoding to a lower resolution or quality level by simply grabbing portions of the image can be very useful for many applications.

Example application for scalable images

Let’s say a server stores a very large, high-resolution image and a client with a smaller display would like to look at and virtually navigate around this image. Because the client’s display resolution is much smaller than the original image resolution and because the bandwidth between the client and server may be limited, the options are to serve the client a small portion of the image in full resolution or the entire image in low resolution. In order to do this, the server would have to extract a portion of the image or extract a low resolution version of the image.

This can be achieved in different ways. If regular image coding is used, then the server would have to decode, process (select an area or downsample the image), and encode the image or transcode it accordingly. On the other hand, if scalable image coding is used, then it is very easy to extract portions of the image in different resolutions. Transcoding to lower resolutions or smaller tiles simply involves extracting the appropriate set of coded data. This requires very little computation, so it allows the server to support simultaneous image streaming sessions for many clients.

Adding security to media

A question that arises is what happens if end-to-end security is required for the application? For example, what if the application requires the image to be encrypted at the source and decrypted only by people who are allowed access? If this is required, then when the media data is transported between the sender and receiver, it must remain encrypted at all times, including when it is stored on the server.

When the media data is encrypted, what happens to the nice property of being able to adaptively stream portions of the high-resolution original image to lower-resolution clients? If the media is encrypted in a media-unaware way, then this property is lost, or the only way to do adaptively stream is by decrypting the image, but this breaks the end-to-end security of the system.

On the other hand, if media-aware security is used, then the security tools can be applied to the media data in a media-aware way in order to preserve the structure of the protected media and allow the server to adaptively stream portions of the protected media data.

JPSEC media-aware security tools

JPSEC was designed to provide media-aware security tools for JPEG-2000 images. It recognizes the structure of the JPEG-2000 image data, and it secures the media data within that structure. Specifically, it recognizes where the media data is located and which parts of the data correspond to which image components (tile, resolution, quality layer, color component, or image subband). It then allows security tools to be applied to subsets of the image data, and specifies the signaling data that must be included in the protected bitstream to allow the protected subsets of data to be extracted. In other words, JPSEC simultaneously allows mid-network transcoding and end-to-end security!

I hope this provides an introduction to JPSEC along with some background and motivation. I’ll dive into more details in future posts! I’ll be evolving this into a publication, so please let me know if you have any comments on the description or the text.

What do you think about JPSEC?

Tags: JPSEC, JPEG 2000, scalable coding, security, image security, media security, media-aware security, media-unaware security, end-to-end security, transcoding, media adaptation, secure transcoding, HP

Please feel free to include a URL in your comments.

The Next-Generation Gaming Experience is immersive, social, mobile, and physical. It raises a number of technology challenges to make it real, but researchers love a challenge! Here is our view of the next-gen gaming experience and a few projects that our researchers are working on to get us there. We showed them at the HP Gaming Summit last week in San Francisco.

There are many press articles and blog posts on the gaming summit. Kate Greene of Technology Review did a great job capturing my thoughts in this article.

Full disclosure: I manage some of these projects and I think they’re cool!

The Next-Generation Gaming Experience

Immersive visual experience with pixels anywhere and pixels everywhere.

Pixels anywhere- including on walls, on floors, on ceilings, on your watch, and on table tops. When pixels are on your coffee table, there is a technology challenge of creating new interaction models to share, control, and interact. When many people are looking at an upright display, there is a shared view of what is up-down-left-right. When many people are sitting around a coffee table, this assumption needs to be revisited, since each person sitting around the table has their own view of up-down-left-right. There is a challenge in inventing new interfaces and interaction models to make sure everyone around the table gets a first class experience. And, there is a challenge in designing applications that work well in this experience.

We showed Misto, our research project that has a touch-screen display embedded into a coffee table. It has applications such as solving jigsaw puzzles, web browsing (including Google earth fly-by’s), and photo sharing, where you can slide photos across the table and spin them to be upright for each person.

Pixels everywhere means that we have flexible pixels, on flat and curved surfaces, with any shape and size. Why limit yourself to a 4:3 or 16:9 aspect ratios? You can use many displays along with real-time video processing algorithms to make displays of any shape, size, and quality. The challenge comes from making many displays, such as projectors, look like one seamless display. Since each individual display has different display characteristics, you need to do some real-time video processing to adapt the color and geometry of the rendered pixels to produce one seamless display.

We showed Panoply, our research project that uses multiple projectors pointed side-by-side on a curved screen to give you an immersive visual experience that covers your entire field of view. We use real-time photometric and geometric calibration algorithms to create one seamless immersive display.

We also showed Pluribus, our research project that uses multiple projectors pointing at one large screen to give you a better-than-theater visual experience. (At the gaming summit we showed Pluribus with 12 projectors, but you can use any number.) We use real-time video processing algorithms to modify the pixels so that they add up to one super-projector, one that has higher image quality in terms of brightness, contrast, resolution, and color quality.

Social experience. It’s a social experience that brings people together and forms communities, with a broader audience than what we have today (maybe including me!). Technologies such as motion sensors are making games easier and more intuitive to play, which will expand the gaming demographics. Also, games will be more integrated with communications such as instant messaging, chatrooms, voice, and video. The challenges lie in integrating these different modalities into one session in the context of a game, and making sure the network priotizes the different types of traffic accordingly. For example, you need very quick response for the game controls, but could delay a half second on a chat message.

Mobile experience. It’s a mobile experience that that lets people play games together on any network, on any device, anywhere in the world. By mobile, we don’t only mean playing a game on your cell phone. We mean that people are mobile- you will game on your TV, on your computer, on your portable player, or on your cell phone- but you still want to have your gaming experience wherever you are. Of course, your experience will be different depending on your device and your connectivity, so your gaming experience may be different in each situation, but it should still be fun!

Challenges come from rendering the game on any device through any network. It is useful to consider different device-network paradigms. For example, if there is enough network bandwidth you could go to an in-network rendering model, where the players view is rendered on a machine in the network, and the resulting video is streamed to the player through a regular video streaming connection. This way, the device only needs to decode a video stream (e.g., an MPEG) rather than have full graphics rendering capabilities. On the other hand, if your device has plenty of graphics capabilities but little network bandwidth, you could go to mode where you only send commands over the network but render the game on the device itself.

Physical experience that maps the physical and virtul worlds. It senses your physical context and triggers experiences accordingly. Your physical context can be your location, your movements, and even your heartrate or who you are next to. Based on your context, it then triggers a multmedia experience. Technology challenges lie in appropriately sensing your context, which requires having sensors that are accurate and power-efficient. Challenges also lie in developing applications and experiences that use context.

We showed Mscape, a.k.a. Mediascapes, our research project aimed at creating and sharing context-aware multimedia experiences. We’ve had Mediascapes deployed around the world, most recently at the Tower of London and Yosemite National Park.

What is your view of the next-gen gaming experience?

What technologies and projects are there out there that will make it real?  (include URLs)

Tags: gaming, next generation gaming experience, HP Labs, VooDoo, HP Gaming Summit, misto, panoply, pluribus, mscape, mediascape, HP

Feel free to include a URL in your comments.