Hot Topics

Home » Security Bloggers Network » An Analysis of the OpenSSL SSL Handshake Error State Security Bypass (CVE-2017-3737)

An Analysis of the OpenSSL SSL Handshake Error State Security Bypass (CVE-2017-3737)

by Fortinet Blog | Latest Posts on January 12, 2018

OpenSSL is a widely used library for SSL and TLS protocol implementation that secures data using encryption and decryption based on cryptographic functions. However, a Security Bypass vulnerability – recently addressed in a patch by the OpenSSL Project –can be exploited to make vulnerable SSL clients or remote SSL servers send clean application data without encryption.

This Security Bypass vulnerability (CVE-2017-3737) is caused by an error when the SSL_read or SSL_write function handles an “error state” during an SSL handshake. In this paper the FortiGuard Labs team examines the root cause of this vulnerability.

The “error state” mechanism was introduced in OpenSSL beginning with version 1.0.2b, It is used to make OpenSSL move into an error state whenever a fatal error occurs during the SSL handshake that would fail if the SSL handshake continued. If SSL_read or SSL_write function is called directly, it checks the SSL handshake state and performs a new SSL handshake automatically if no handshake has been initiated. If a fatal error occurs during the SSL handshake, OpenSSL moves into the error state and returns an error message to the caller. However, the problem occurs if the caller doesn’t check the error state and simply calls the SSL_read or SSL_write function again, because it then sends application data without encryption.

The following code snippet was taken from OpenSSL 1.0.2m. (Comments added by me have been highlighted.)

ssl/s3_pkt.c:

Sponsorships Available

638 int ssl3_write_bytes(SSL *s, int type, const void *buf_, int len)

639 {

640 const unsigned char *buf = buf_;

641 int tot;

642 unsigned int n, nw;

643 #if !defined(OPENSSL_NO_MULTIBLOCK) && EVP_CIPH_FLAG_TLS1_1_MULTIBLOCK

644 unsigned int max_send_fragment;

645 #endif

646 SSL3_BUFFER *wb = &(s->s3->wbuf);

647 int i;

648

649 s->rwstate = SSL_NOTHING;

650 OPENSSL_assert(s->s3->wnum <= INT_MAX);

651 tot = s->s3->wnum;

652 s->s3->wnum = 0;

653

654 if (SSL_in_init(s) && !s->in_handshake) { //checks to see if the SSL handshake state is initiated. The state will be SSL_ST_INIT the first time.

655 i = s->handshake_func(s); //performs a new SSL handshake if no handshake has been initiated.

656 if (i < 0)

657 return (i);

658 if (i == 0) {

659 SSLerr(SSL_F_SSL3_WRITE_BYTES, SSL_R_SSL_HANDSHAKE_FAILURE);

660 return -1;

661 }

662 }

ssl3_write_bytes() is called by the SSL_write function to send the application data. It checks the SSL handshake state and performs the SSL handshake if needed.

ssl/s3_clnt.c:

898 int ssl3_get_server_hello(SSL *s)

899 {

900 STACK_OF(SSL_CIPHER) *sk;

901 const SSL_CIPHER *c;

902 CERT *ct = s->cert;

903 unsigned char *p, *d;

904 int i, al = SSL_AD_INTERNAL_ERROR, ok;

….

1077 if (i < 0) {

1078 /* we did not say we would use this cipher */

1079 al = SSL_AD_ILLEGAL_PARAMETER;

1080 SSLerr(SSL_F_SSL3_GET_SERVER_HELLO, SSL_R_WRONG_CIPHER_RETURNED);

1081 goto f_err; //a fatal error occurs

1082 }

….

1170 f_err:

1171 ssl3_send_alert(s, SSL3_AL_FATAL, al); //sends an SSL alert packet

1172 err:

1173 s->state = SSL_ST_ERR; //moves into an error state

1174 return (-1);

1175 }

We used a vulnerable SSL client as a target during the test. During the SSL handshake it received a malformed server hello message from the SSL server controlled by the attacker. ssl3_get_server_hello() is called to handle this server hello message and a fatal error occurs, causing OpenSSL to move into an error state by setting s->state from SSL_ST_INIT to SSL_ST_ERR.

ssl/s3_pkt.c:

638 int ssl3_write_bytes(SSL *s, int type, const void *buf_, int len)

639 {

640 const unsigned char *buf = buf_;

641 int tot;

642 unsigned int n, nw;

643 #if !defined(OPENSSL_NO_MULTIBLOCK) && EVP_CIPH_FLAG_TLS1_1_MULTIBLOCK

644 unsigned int max_send_fragment;

645 #endif

646 SSL3_BUFFER *wb = &(s->s3->wbuf);

647 int i;

648

649 s->rwstate = SSL_NOTHING;

650 OPENSSL_assert(s->s3->wnum <= INT_MAX);

651 tot = s->s3->wnum;

652 s->s3->wnum = 0;

653

654 if (SSL_in_init(s) && !s->in_handshake) { // SSL_in_init is called again to check the state

If the vulnerable SSL client doesn’t check the error state and call SSL_write function to send application data again, ssl3_write_bytes() is called and uses SSL_in_init() to check the handshake state again.

include/openssl/ssl.h:

1749 # define SSL_in_init(a) (SSL_state(a)&SSL_ST_INIT) //s->state is now SSL_ST_ERR, and check returns are false.

This time the check fails, the SSL handshake is bypassed and the application data will be sent without encryption.

The following traffic dump shows how the clean application data is sent:

traffic

During this attack, the attacker entices the vulnerable SSL client to connect to a malicious SSL server. The SSL client may bypass the handshake process and send the application data without encryption. The SSL server may also have the same vulnerability if SSL_read or SSL_write function is called directly.

NOTE: authentication is NOT required to exploit this vulnerability.