Issue summary: Receiving a DTLS record for a future epoch while a handshake is in progress causes OpenSSL to buffer far more memory than the record it...

Issue summary: Receiving a DTLS record for a future epoch while a handshake
is in progress causes OpenSSL to buffer far more memory than the record
itself requires.

Impact summary: A peer can use a small amount of network traffic to make an
OpenSSL DTLS endpoint retain a disproportionately large amount of memory,
which may lead to a Denial of Service.

CWE: CWE-405: Asymmetric Resource Consumption (Amplification)

Description: While a DTLS handshake is in progress, a peer may legitimately
have already moved on to the next epoch (for example, having sent its
ChangeCipherSpec and Finished messages) before the local endpoint has
processed the same transition, typically because of reordering on the
underlying UDP transport. OpenSSL buffers such early records so that they
can be processed once the local endpoint catches up.

Buffering a record currently retains the entire read buffer it arrived in,
which is sized to hold the largest possible DTLS record (around 16
kilobytes), rather than just the bytes that make up the record itself. Up
to 100 such records may be buffered per connection. As a result, a peer
that sends a stream of small forged records claiming to belong to the next
epoch can cause an OpenSSL DTLS endpoint to retain around 1.7 megabytes of
memory, despite sending only a small fraction of that amount of data over
the network.

An attacker therefore gains a memory amplification factor of around 1200,
and can multiply the effect across as many associations as it is ab…