Untrusted data can lead to DoS attack due to hash collisions and stack overflow in MessagePack

Impact

When this library is used to deserialize messagepack data from an untrusted source, there is a risk of a denial of service attack by either of two vectors:

1. hash collisions - leading to large CPU consumption disproportionate to the size of the data being deserialized.
2. stack overflow - leading to the deserializing process crashing.

Patches

The following steps are required to mitigate this risk.

1. Upgrade to a version of the library where a fix is available
2. Add code to your application to put MessagePack into the defensive UntrustedData mode.
3. Identify all MessagePack extensions that implement IMessagePackFormatter implementations that do not ship with the MessagePack library to include the security mitigations. This includes those acquired from 3rd party packages and classes included directly into your project. Any AOT formatters generated with the MPC tool must be regenerated with the patched version of mpc.
4. Review your messagepack-serializable data structures for hash-based collections that use custom or unusual types for the hashed key. See below for details on handling such situations.

Review the MessagePackSecurity class to tweak any settings as necessary to strike the right balance between performance, functionality, and security.

Specialized IEqualityComparer implementations provide the hash collision resistance.
Each type of hashed key may require a specialized implementation of its own.
The patched MessagePack library includ…