SSL/TLS Secure Channel - SCHANNEL - Troubleshooting

When encrypted communication is enabled in YSoft SafeQ 6, the subsystems connect to each other using strong protocols or protocols defined in the underlying operating system. You may encounter some issues when your server runs on an out-of-date OS or when the OS is up to date but your MFD has old firmware. This guide should help you identify and fix problems related to security protocols.

Terminal/MFD Cannot Be installed, the Error "Could not create SSL/TLS secure channel" Appears in the terminalserver.log Log File

This is a general error message meaning that an error occurred during SSL/TLS cipher negotiation, the data encryption/decryption process or the data was corrupted.

The following sections describe several cases that may cause this message to occur.

The Device Only Supports Weak DH Keys

When the device and YSoft SafeQ 6 Server agree on the cipher suite containing a Diffie-Hellman key exchange (DH or DHE) algorithm and the DH keys provided by the device are shorter than Windows' default 1024 bits, Terminal Server, which inherits the security settings from the underlying operating system, it may refuse a connection to such a device.

Conditions/environment

YSoft SafeQ 6 Server (Terminal Server) is running on the Windows Server OS machine with one of the following updates installed:

  • June 2016 update rollup

    • KB3161608 (Windows 7, Windows Server 2008 R2)

    • KB3161606 (Windows RT 8.1, Windows 8.1, Windows Server 2012 R2)

  • July 2016 update rollup

    • KB3172605 (Windows 7, Windows Server 2008 R2)

    • KB3172614 (Windows RT 8.1, Windows 8.1, Windows Server 2012 R2)

  • June 2016 cumulative update

    • KB3163018 (Windows 10, Windows Server 2016 Technical Preview 4)

How to recognize this case

Try to connect to the MFD interface via the web browser on the same server as Terminal Server is running. Is the browser able to connect?

  • Internet Explorer – usually just the general failure appears: "This page can’t be displayed"

  • Mozilla Firefox – a connection is denied with the error code SSL_ERROR_WEAK_SERVER_EPHEMERAL_DH_KEY

Scan the device using the following nmap command:

nmap --script ssl-enum-ciphers -p DeviceSecurePort DeviceAddress

A list of SSL/TLS protocol versions and cipher suites supported by the device similar to the following picture should display.

images/download/attachments/160483253/DH_-_512_-_04.png

In parentheses after the DHE cipher suites is a DH key length. If this number is lower than 1024, you may have encountered this case.

How to fix this case

On Windows servers, this can only be avoided by setting the required DH key length lower using registers. You can add the registry key manually or use the following fix:

  1. Create a dh.reg file with the following content:

    Windows Registry Editor Version 5.00
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\Diffie-Hellman]
    "ClientMinKeyBitLength"=dword:00000200
  2. Open this file in the machine with Terminal Server running. Windows should automatically create the given registry key.

  3. Try to install the device again.

Do not set a lower length of DH keys unless really necessary. Supporting DH keys of lower than 1024 bits makes your machine vulnerable to some attacks.

The Incorrect Implementation of the DH Algorithm in the Device

When using a cipher suite containing the Diffie-Hellman key exchange (DH or DHE) algorithm, occasionally, a device detects a fatal error resulting in an SSL/TLS session termination.

Conditions/environment

YSoft SafeQ 6 (Terminal Server) is running on the Windows Server OS machine with one of the following updates installed:

  • June 2016 update rollup

    • KB3161608 (Windows 7, Windows Server 2008 R2)

    • KB3161606 (Windows RT 8.1, Windows 8.1, Windows Server 2012 R2)

  • July 2016 update rollup

    • KB3172605 (Windows 7, Windows Server 2008 R2)

    • KB3172614 (Windows RT 8.1, Windows 8.1, Windows Server 2012 R2)

  • June 2016 cumulative update

    • KB3163018 (Windows 10, Windows Server 2016 Technical Preview 4)

How to recognize this case

The installation of a Konica Minolta device randomly crashes on different installation steps when trying reinstallation several times.

How to fix this case

On Windows servers, this can only be avoided by disabling the DH key exchange algorithm using registers. You can add the registry key manually or use the following fix:

  1. Create a dh.reg file with the following content:

    Windows Registry Editor Version 5.00
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\KeyExchangeAlgorithms\Diffie-Hellman]
    "Enabled"=dword:00000000
  2. Open this file in the machine with Terminal Server running. Windows should automatically create the given registry key.

  3. Try to install the device again.

This registry modification disables DH/DHE ciphers for the whole machine, thus, no communication links dependent on the operating system settings will be able to use them.

The MFD Only Supports Older SSL/TLS Protocol Versions, but also Accepts Higher Ones

YSoft SafeQ 6 Server needs to connect to the device during terminal installation. YSoft SafeQ Server and the device negotiate the SSL/TLS protocol version they will be communicating with. During this negotiation, the parties always try to connect using the highest protocol version both of them support. The problem here is, for example, when YSoft SafeQ Server supports TLSv1.2 and the device does not but pretends it does, they agree on this version and the connection fails.

How to recognize this case

Try to connect to the MFD interface via a web browser (with TLSv1.2 enabled) on the same server as Terminal Server is running on. Is the browser able to connect?

  • Mozilla Firefox – the connection is denied with the error code SSL_ERROR_BAD_MAC_ALERT

  • Google Chrome – the connection is denied with the error code ERR_SSL_BAD_RECORD_MAC_ALERT

How to fix this case

In the YSoft SafeQ 6 management interface, go to System settings (Expert options) and search for the property securityProtocolTypesForOutboundCommunication. Set this system property to only the SSL/TLS protocol versions supported by your MFD.

Be careful when modifying this property. Some other devices may stop communicating when the SSL/TLS protocol versions are not set properly.

Communication with YSoft SafeQ Payment System Fails with the Error "Could not create SSL/TLS secure channel" in the terminalserver.log Log File

Older Windows servers do not support cooperation of TLSv1.2 and certificates using the SHA-512 hash function. This causes failures in the connection to YSoft SafeQ Payment System.

Conditions/environment

YSoft SafeQ Server (Terminal Server) is running on the Windows Server OS machine without one of the following updates installed:

  • August 2014 update rollup

    • KB2973337 (Windows 7, Windows Server 2008 R2)

    • KB2975331 (Windows RT 8, Windows 8, Windows Server 2012)

    • KB2975719 (Windows RT 8.1, Windows 8.1, Windows Server 2012 R2)

How to recognize this issue

Try to connect to Payment System from the Terminal Server machine via Internet Explorer with TLSv1.2 enabled. Is it able to connect?

  • Internet Explorer – the error message is displayed: "Make sure TLS and SSL protocols are enabled."

Be aware that this is only for troubleshooting purposes. YSoft SafeQ no longer supports Internet Explorer.


How to fix this issue

If you encounter this issue with TLSv1.2 enabled on the machine with Terminal Server running (in the registry or in the securityProtocolTypesForOutboundCommunication expert system property), apply the following update to this machine https://support.microsoft.com/en-us/kb/2973337 .

The Device Supporting Only Higher Versions of the SSL/TLS Protocol Cannot Connect to YSoft SafeQ 6 Server

Older Windows servers do not have higher versions of SSL/TLS protocol enabled. Terminal Server inherits these settings from the system, and, thus, devices only supporting higher versions cannot connect.

Conditions/environment

YSoft SafeQ 6 Server (Terminal Server) is running on the Windows Server OS machine with one of the following systems:

  • Windows Server 2008 R2 with KB3080079 installed

  • Windows Server 2012

  • Windows Server 2012 R2

How to recognize this issue

Scan the machine with Terminal Server running using the following nmap command:

nmap --script ssl-enum-ciphers -p 5012 MachineAddress

A list of SSL/TLS protocol versions and cipher suites supported by Terminal Server should display.

Scan the device using the nmap command:

nmap --script ssl-enum-ciphers -p DevicePort DeviceAddress

A list of SSL/TLS protocol versions and cipher suites supported by the device should display.

If the protocols supported by the device and the protocols supported by Terminal Server are exclusive lists, you have encountered this issue.

How to fix this issue

On Windows servers, this can only be avoided by setting the required SSL/TLS protocol versions using registers. You can add the registry keys manually or use the following fix:

  1. Create a tls.reg file with the following content:

    Windows Registry Editor Version 5.00
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL 3.0\Server]
    "Enabled"=dword:00000000
    "DisabledByDefault"=dword:00000001
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.0\Server]
    "Enabled"=dword:00000000
    "DisabledByDefault"=dword:00000001
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.1\Server]
    "Enabled"=dword:00000001
    "DisabledByDefault"=dword:00000000
    [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.2\Server]
    "Enabled"=dword:00000001
    "DisabledByDefault"=dword:00000000

    These values will disable support for SSLv3 and TLSv1.0 and enable TLSv1.1 and TLSv1.2. Of course, you can modify them before proceeding to the next step according to your requirements.

  2. Open this file is in the machine with Terminal Server running, Windows should automatically create the given registry keys.

  3. Restart the machine with Terminal Server.

The Device Is Installed Correctly, but Cannot Connect to YSoft SafeQ 6 Server

This usually happens when the device is not able to verify the YSoft SafeQ 6 Server (Terminal Server) certificate or the server certificate uses a cryptographic algorithm not supported by the device.

The Certificate Is Not Trusted

By default, the communication between the terminal and Terminal Server is encrypted using the same default certificate for all YSoft SafeQ 6 deployments. This is not trusted in the MFDs, so a new key/certificate needs to be created and Terminal Server configured to use it (follow the chapter Configuring secured connection between terminals and Terminal Server). Also, the certificate must be imported to the device as trusted.

The Device Does Not Support the Cryptographic Algorithm Used for the Certificate or the Key Length

Since YSoft SafeQ 6 MU17 (and newer builds), the default certificate in the Terminal Server uses the algorithm SHA-256 (instead of the previous SHA-1) and has an RSA key of 2048 bits long.

Some older devices do not support either the SHA-256 algorithm or RSA keys of a maximum length of 1024 bits. When it really is a must to communicate in an encrypted way with such a device, a new key/certificate compatible with it needs to be created. For more information about the Terminal Server certificate setting see Configuring secured connection between terminals and Terminal Server .