Designing Windows Store Apps for Performance

Handling media

Including media in your app is a compelling way to enhance the user experience. XAML makes it easy to include video, audio, and images in your application. Just pick the right UI element, set the source, and voila, XAML and WinRT take care of all the heavy lifting of decoding and presenting the media appropriately.

For example, to include video in your app, you just drop a MediaElement on your user interface and set the source to a local or remote video. Listing 3-14 shows the simple XAML you need to write to do this. Using audio and images are equally straightforward.

LISTING 3-14 Including media in your app is simple.

<MediaElement x:Name="mediaSimple"
              Source="Videos/video.mp4"
              Width="400" />

While including media in your app is deceptively easy, you can do a couple of things to ensure proper and efficient playback. In the following sections, I go through a number of things to keep in mind when handling media content:

• Playing video
• Displaying images
• Playing audio
• Releasing resources

Playing video

WinRT supports a number of popular video formats. Selecting the right format can have a great impact on both performance and power utilization. For videos, the recommended encoding is H.264, and for local playback, the preferred file container is MP4. H.264 decoding is accelerated through most recent graphics hardware. Although hardware acceleration for VC-1 decoding is broadly available, for a large set of graphics hardware on the market, the acceleration is limited in many cases to a partial acceleration level, rather than a full-steam-level hardware offload.

Apart from picking the optimal format, there are a number of things to keep in mind when playing video. You should prefer videos that are scaled to the display resolution. Decoding video takes a lot of memory and GPU cycles, so choose a video format close to the resolution it will be displayed at. There is no point in using the resources to decode high-definition video if it’s going to get scaled down to a much smaller size. If the back end offers multiple formats, make sure to pick the proper format.

Defer processing of media content

By default, MediaElement retrieves one of the initial frames to display before the video is played. Opening the file, decoding the frames, and picking an image is not a trivial operation. If the user never plays the file, this is a lot of work to do up front. Fortunately, MediaElement has a PosterSource property that allows you to specify an image that can be used instead. Note, however, that this property is useful only if Source is not set in the markup as well. To use PosterSource, set it to some image and then dynamically set the Source as needed at a later point. The PosterSource image is displayed while the work to retrieve an appropriate frame from the video is in progress. Once a frame has been identified, it is cached to reduce the necessary work the next time the video is used. The PosterSource is also displayed if the MediaFailed event fires.

If your app displays a collection of videos, the use of PosterSource can have a significant effect on the user experience because XAML doesn’t have to retrieve and display an image from each of the files. Furthermore, if your app handles multiple videos, you should defer setting the source until needed. When you set the source, XAML does work to prepare the video for playback. If you have many videos, that’s a significant amount of work, and chances are that most of it will not be needed because the user might not play all the videos. By deferring this work, the app can present the users with the available options much faster.

Controlling playback

In Windows 8, the MediaElement tag didn’t provide any controls to manage the playback of video or audio by default, so you had to implement those yourself. If you want to customize the controls, this approach is still viable, but for all those cases where you just want to play back media content, it is a little tedious to implement this. Fortunately, the AreTransportControlsEnabled property was added to MediaElement in Windows 8.1. If you set this to true, you get a set of standard media controls.

The controls include everything you need for controlling regular playback of both audio and video content. More importantly, if you enable AreTransportControlsEnabled, the MediaElement automatically does the right thing in many cases. For instance, the controls are displayed on demand and automatically removed when no longer used. This optimizes the playback of video because XAML doesn’t have to composite the controls on top of the video frames.

Full-screen playback

If video playback is an important part of your application, you should prefer full-screen playback over embedded playback. Full-screen playback enjoys several optimizations that provide for a better user experience as well as better power utilization. In Windows 8, you had to make sure the size of the MediaElement matched the display resolution. In Windows 8.1, you can use the IsFullWindow property on MediaElement. If you set this to true, XAML makes sure your video is played at the proper resolution and you don’t have to set the dimensions on the MediaElement.

To further optimize playback, make sure no elements are rendered on top of the MediaElement. If you’re implementing your own playback controls, make sure they are removed when no longer needed. Rendering elements on top of the MediaElement forces composition to happen, which adds a bit of overhead and prevents some internal optimizations. If you’re using AreTransportControlsEnabled, you don’t have to do anything else because it automatically removes the controls as needed.

A common problem in this regard is progress indicators. Retrieving media content can take some time, so showing a progress indicator can be a good idea to let the user know that the app is working on retrieving content. However, if you do that, you must make sure to disable the progress indicator by setting the IsActive property to false once the media is ready. If you don’t do that, the progress indicator continues to run its animation even if it isn’t visible. That hurts performance and reduces battery life significantly because WinRT cannot apply its optimizations in this area. If the media is not full screen, this also causes overdraw.

A good way to detect this problem is to enable DebugSettings.EnableRedrawRegions and run the app. This causes the Desktop Window Manager (DWM) to display sections in different colors each time it updates the display. If you have a progress indicator working in the background, you’ll see the DWM compositing the dots along with the video.

One thing to be aware of when using EnableRedrawRegions is that this works on the DWM level, so you might see other applications trigger DWM work as well. Somewhat surprisingly, the artifacts of this work show up even though your app uses the entire screen. If you see any DWM updates you didn’t expect, this might be caused by other applications. To reduce the noise, close or minimize all other applications while you test with this property enabled.

Embedded playback

For some apps, full-screen video is not really an option. If your app edits video, you might want to show controls and data alongside the media content. If the media content is not available in a decent resolution, showing it in full screen leaves a lot to be desired. Whatever the reason, there are scenarios where displaying full-screen video is not the way to go. For those cases, you can embed the MediaElement on the page.

If you embed media on the page, WinRT is not able to apply all the optimizations available for full-screen display. However, the same tips still apply. To optimize performance of the playback, you don’t want to place any XAML elements on top of your MediaElement. If you implement your own media controls, make sure to place these next to the MediaElement instead of on top as illustrated in Figure 3-7. If you do place them on top, make sure to remove them after a while, similar to what AreTransportControlsEnabled does.

Displaying images

Images often play an important part of the presentation in Windows Store apps. As discussed in Chapter 2, handling images is far from trivial, and the amount of work needed to handle an image increases with the size of the image.

The optimal approach is to make sure all your image resources are stored in the size needed. Using correctly sized images affects performance noticeably in many cases. If the images are downloaded dynamically, smaller images load faster. Smaller images are also decoded and transferred to the GPU faster, so there are several reasons why getting the size right is important.

If you cannot specify the native size of the images, the next best thing to do is to make sure your app decodes the images in the proper size. If you’re using the BitmapImage class, you can use the DecodePixelWidth and DecodePixelHeight properties to specify the decode size. This affects decoding itself and the amount of data transferred to the GPU, both of which can affect performance significantly.

Listing 3-15 is not very effective because it forces the entire image to be loaded, decoded, and then resized. This increases memory usage significantly and might exhaust GPU bandwidth as well.

LISTING 3-15 The ineffective way to handle a large image—don’t do this.

<!-- Don't do this -->
<Image Source="ms-appx:///Assets/highresPicture.jpg" Width="300" Height="200"/>

Listing 3-16 does the right thing by specifying the height and width for the decoded image. This reduces the size of the decoded image and thus the memory usage, and it decreases the amount of data sent to the GPU. Be sure to either scale your images to the desired size or set the decode size.

LISTING 3-16 The proper way to handle a large image.

<Image>
    <Image.Source>
        <BitmapImage UriSource="ms-appx:///Assets/highresPicture.jpg"
                     DecodePixelWidth="300" DecodePixelHeight="200"/>
    </Image.Source>
</Image>

Also, to prevent images from being decoded more than once, assign the Image.Source property from an Uri rather than using a memory stream. The XAML engine can associate multiple Bitmaps based on the same Uri with a single decoded image. There’s no similar optimization for memory streams, so if you use a Bitmap based on a memory stream as the source of multiple images, XAML has to decode the image multiple times.

Specifying DecodePixelWidth and DecodePixelHeight is useful for handling images efficiently. However, if all you really need are thumbnails, the StorageFile.GetThumbnailAsync is a better option because it can read the thumbnails cached by the file system. Listing 3-17 shows how to use GetThumbnailAsync to retrieve thumbnails of images in a folder.

LISTING 3-17 Using GetThumbnailAsync to retrieve a thumbnail of an image file and assign it as the source of an image in XAML.

var picker = new FileOpenPicker();
picker.FileTypeFilter.Add(".jpg");
picker.FileTypeFilter.Add(".jpeg");
picker.FileTypeFilter.Add(".png");
picker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;

var file = await picker.PickSingleFileAsync();

var fileThumbnail = await file.GetThumbnailAsync(ThumbnailMode.SingleItem, 64);

var bmp = new BitmapImage();
bmp.SetSource(fileThumbnail);

image.Source = bmp;

Listing 3-17 creates an instance of FileOpenPicker, adds some commonly used image formats, and asks the user to pick a file. GetThumbnailAsync is then used to create a StorageItemThumbnail with a requested size of 64 for the picked file.

Using thumbnails is appropriate for apps that let the user browse any kind of media. GetThumbnailAsync can retrieve thumbnails for images, audio files, videos, documents, and even folders containing these types by specifying the proper ThumbnailMode. If your app needs only thumbnails, this approach is preferable to using DecodePixelWidth and DecodePixelHeight.

Playing audio

WinRT also supports numerous audio formats, but for the best results you should prefer Advanced Audio Coding (AAC) and MP3, because they offer the best performance. If your app plays short, low-latency sound effects, use WAV files with uncompressed Pulse-Code Modulation (PCM) data to reduce processing overhead that is typical for compressed audio formats.

When playing sound, it might make sense to turn off the display after a while. However, if your app shows information related to the playback, you can instruct Windows to leave the display on as described in the upcoming “Display deactivation” section. Remember to call DisplayRequest.RequestRelease if the playback stops for some reason, so your app doesn’t keep the display needlessly on.

Windows also supports hardware offload of audio playback. To enable this, your app must set MediaElement.AudioCategory to either ForegroundOnlyMedia or BackgroundCapableMedia. Hardware audio offload optimizes audio rendering, which can improve both functionality and battery life.

Releasing resources

Media files can be huge, and consequently accessing them increases the memory usage of your app significantly. To use media content efficiently, you must release the resources as soon as possible when you’re done using them.

Streams

An obvious example of resources is streams. Streams can be used to read and write media content if you want to do more than the basics provided by the media UI elements. For instance, your app could read media content from a stream in order to modify it. If you read or write media content through a stream, make sure to close the stream as soon as the app is done reading from it.

Listing 3-18 shows an example where an app reads a picture from a file, modifies the content, and writes the updated picture to the stream of a bitmap. Notice how the streams are used inside using blocks. This, of course, implicitly calls the Dispose method at the end of the block and thus releases the resources as necessary.

LISTING 3-18 Reading and writing media content via streams.

// Get picture from library
var folder = KnownFolders.PicturesLibrary;
var file = await folder.GetFileAsync("pic.png");

// Read stream from file
var streamRef = RandomAccessStreamReference.CreateFromFile(file);
using (var fileStream = await streamRef.OpenReadAsync())
{
    // Decode format
    var decoder = await BitmapDecoder.CreateAsync(fileStream);
    var frame = await decoder.GetFrameAsync(0);

    // Get pixels as byte array
    var pixelDataProvider = await frame.GetPixelDataAsync();
    var pixels = pixelDataProvider.DetachPixelData();

    // Change picture by manipulating the byte array
    ManipulatePixels(pixels);

    // Create bitmap and write the modified pixels
    var bitmap = new WriteableBitmap((int)frame.PixelWidth, (int)frame.PixelHeight);
    using (var stream = bitmap.PixelBuffer.AsStream())
    {
        stream.Write(pixels, 0, pixels.Length);
    } // this closes the stream

    // Display modified picture
    image.Width = frame.PixelWidth;
    image.Height = frame.PixelHeight;
    image.Source = bitmap;
} // this closes the file stream

Display deactivation

If you leave your device idle for a while, Windows dims and eventually turns off the display to preserve power. That’s rarely the desired behavior when playing media content such as video. The user should be able to watch videos without having to tap the display every now and then. Yes, this sounds silly, but believe me I have seen media apps that didn’t do this.

To prevent Windows from turning off the display during media playback, your app must call the DisplayRequest.RequestActive method. This informs Windows it should not turn off the display to preserve power. That ensures the user can enjoy the media playback uninterrupted, which is exactly what you want as long as the user is actually watching the media.

If the user pauses the media or watches the content to the end, or the app encounters a media playback error, the display is no longer required and your app should call DisplayRequest.RequestRelease to let Windows know that the display can be turned off again when idle. If your app doesn’t do that, the display will drain the device’s battery, and possibly leave the user with an unusable device.

If your app uses AreTransportControlsEnabled as described earlier and the playback is full screen, or if your app sets the IsFullWindow property, XAML automatically calls DisplayRequest.RequestActive and DisplayRequest.RequestRelease as appropriate.