Remote Desktop Protocol (RDP) 10 AVC/H.264 improvements in Windows 10 and Windows Server 2016 Technical Preview

All postings, articles, and other content on this blog are provided "AS IS" with with no warranties, and confer no rights. Any code, demo, or sample on this blog is subject to the terms specified at http://www.microsoft.com/info/cpyright.mspx.

Hello Everyone, this is Jeroen van Eesteren from the Remote Desktop team. I am writing you today to let you know about the graphics remoting enhancements we made to the Remote Desktop Protocol (RDP) version 10 which shipped with the Windows 10 1511 update and the Windows Server 2016 Technical Preview 4, both released in November 2015. But first, let me provide some background and a brief history on why we made these improvements.

Graphics Remoting challenges

Monitor resolutions continue to significantly increase; today larger resolutions such as 1440p and 4K are common. The increase in resolution sizes present a challenge for graphics remoting protocols as there are more pixels which need to be encoded on the server and more bandwidth is required to transfer these encoded pixels over the network. In addition, the demand for rich and fluent graphics experience in remoting scenarios increased due to more graphics intensive applications being used. For example, with Windows Server 2016 Technical Preview we now enable OpenGL applications with RemoteFX vGPU scenarios which enables support for additional high-end engineering design applications that use OpenGL.

RDP and AVC/H.264

The graphics compression (codec) standard which has been used by RDP for a couple of releases is the ITU-T H.264 codec (also known as MPEG-4 AVC (Advanced Video Coding)). The benefit of this codec is that it is widely available in hardware so that CPU intensive encoding and decoding can be offloaded to a separate hardware block.

With RDP 8, we introduced RemoteFX Media Streaming which uses AVC/H.264 to remote video content detected on the server. RemoteFX Media Streaming replaced Multi Media Redirection (MMR). Note: MMR is now completely removed from RDP 10 given that RemoteFX Media Streaming works for all types of video content whereas MMR which just worked for some. More details can be found here.

With RDP 8.1 we introduced an AVC/H.264 mixed mode which in addition to using RemoteFX Media Streaming, extended support for AVC/H.264 to images as well, while text is compressed using a proprietary Codec. This mode is used by Windows RT devices running Windows 8.1 and some 3rd party RDP implementations.

With RDP 10 we are now taking AVC/H.264 support a step further with the introduction of full-screen AVC 444 mode.

Introducing the AVC 444 mode

The main challenge to use AVC/H.264 as the one and only Codec in Remote Desktop scenarios is that text shows a halo effect with typical implementations of AVC/H.264. This is caused by the color conversion process that happens as part of the compression which throws away some of the

chrominance information, as represented in the 4:2:0 format. To the human eye the lack of chrominance information is not as apparent with video content, however with Remote Desktop scenarios, where mostly text is used, it is something that is noticeable and users will perceive this as blurry. The AVC/H.264 standard defines the capability to use 4:4:4 format which doesn't lose the chrominance during conversion, however typically this isn't part of most AVC/H.264 hardware encoder and decoder implementations and thus provides a challenge. To show the difference between 4:4:4 and 4:2:0 please see the following image which shows easily noticeable differences:

As part of the AVC 444 mode in RDP 10 we solved the challenge to get 4:4:4 quality text with 4:2:0 hardware encoders / decoders. In addition, with the AVC 444 mode we were able to improve the frame throughput significantly, for example with 1440p we can achieve a consistent frame rates of up to 50 fps on standard hardware.

AVC 444 and AVC/H.264 Hardware Encoders / Decoders

With the Windows Remote Desktop Client (MSTSC.EXE) the AVC 444 mode automatically uses the AVC/H.264 Hardware decoder if available via the Windows DirectX Video Acceleration (DXVA) API. In addition, the GPU must be DirectX 11.0 capable and the H.264 hardware decoder must support Level 4.1 and BT. 709 Full Range color conversion. If the resolution of the remote session exceeds the capability of the hardware decoder, software decoding will be used instead.

The RDP server can take advantage of any AVC/H.264 hardware encoder exposed in Windows via Media Foundation Transform and is therefore not hardware vendor dependent.

In which RDP 10 Remote Desktop scenarios is the AVC 444 mode available?

Today, the AVC444 mode is enabled by default for all RemoteFX vGPU scenarios which use RDP 10 on Windows Server 2016 Technical Preview 4. The built-in Windows 10 Remote Desktop client (MSTSC.EXE) is currently the only client that has support for AVC444 but we are planning to add this mode to our other remote desktop clients for Mac OS X, Android, iOS, Windows Universal Windows Platform (UWP) and Windows Phone in the future as well.

What about Windows N and KN SKU's where the AVC/H.264 codec is not available?

To allow use of the AVC/H.264 codec on the Windows 10 N and KN you have to download the Media Feature Pack for N and KN versions: https://www.microsoft.com/en-us/download/details.aspx?id=48231

How to enable the AVC444 mode with RDP 10 in non-RemoteFX vGPU scenarios

With the Windows 10 1511 update and the Windows Server 2016 Technical Preview 4 we added two new group policies, which can be found in the Group Policy Editor (gpedit.msc) under Computer Configuration -> Administrative Templates -> Windows Components -> Remote Desktop Session Host -> Remote Session Environment:

1. Prioritize H.264/AVC 444 Graphics mode for Remote Desktop connections

When enabled on the RDP Server, the H.264/AVC 444 mode will be prioritized when the RDP 10 client and server both support AVC/H.264 and support the AVC 444 mode. Note: For Remote Desktop Session Host (RDSH) environments only full desktop sessions are supported with H.264/AVC 444, RemoteApp sessions still use the proprietary codecs for now.

2. Configure H.264/AVC hardware encoding for Remote Desktop connections

This policy lets you enable hardware encoding for AVC/H.264, when used in conjunction with the AVC444 mode. When enabled, each remote desktop monitor will use up one AVC/H.264 encoder on the server. If all AVC/H.264 encoders are in use, the RDP Server will automatically fallback to using Software.

How to determine if a Remote Desktop session is using AVC444 mode and is using hardware encoding

RDP logs an event to the eventlog which helps to determine if you are running in the AVC 444 mode and whether Hardware Encoding is used:

Launch the Event Viewer in Windows on the RDP server and navigate to the following node:

Applications and Services Logs -> Microsoft -> Windows -> RemoteDesktopServices-RdpCoreTS

To determine if AVC 444 mode is used, look for Event ID 162, if "AVC Available: 1 Initial Profile: 2048" than AVC 444 is used.

To determine if Hardware Encoding is used, look for Event ID 170, if "AVC hardware encoder enabled: 1" than hardware is used

Enjoy!

Note: Questions and comments are welcome. However, please DO NOT post a request for troubleshooting by using the comment tool at the end of this post. Instead, post a new thread in the RDS & TS forum. Thank you!

Cool insight, thanks for the clear article!

Does the Hyper-V virtual machine connection also make use of these enhancements?

If not, is this something that is planned for a future update?

Source: Remote Desktop Protocol (RDP) 10 AVC/H.264 improvements in Windows 10 and Windows Server 2016 Technical Preview