CVE-2026-17507
HighCVSS 8.7Summary
In the MLS implementation (RFC 9420) of Bouncy Castle for Java before 1.86, the uint32 leaf_index is stored as a signed int, allowing a wire value with the top bit set to decode as a negative number. Incorrect comparison in GroupKeySet.SecretTree.hasLeaf and Group.validateRemove allows bypassing membership checks, potentially leading to a DoS attack via unbounded growth of the node list and JVM heap exhaustion.
Risk Assessment
A single small message from a group member can cause a denial of service for all other members of the MLS group, causing memory exhaustion and application hang. This affects group communication availability.
Recommendation
Update Bouncy Castle for Java to version 1.86 or later. The new version interprets the value as unsigned via Integer.toUnsignedLong, rejecting out-of-range senders.
Other vulnerabilities in Bouncy Castle for Java
See all- CVE-2026-18036High
A side-channel vulnerability exists in the NTRU implementation of Bouncy Castle for Java before version 1.86. The reduction of secret values using the % operator in three helper functions causes execution time to depend on the secret operand (division by a variable), potentially allowing an attacker to recover information about the NTRU private key through timing measurements.
- CVE-2023-33201Medium
Bouncy Castle for Java before version 1.74 is affected by an LDAP injection vulnerability. This vulnerability affects applications that use an LDAP CertStore from Bouncy Castle to validate X.509 certificates, where the certificate's Subject Name is inserted into an LDAP search filter without proper escaping.
Original NVD description (English source)
In Bouncy Castle for Java before 1.86, the MLS implementation (org.bouncycastle.mls) holds RFC 9420's uint32 leaf_index in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input, and it must still decode, since the MLS interop test vectors round-trip the full range. GroupKeySet.SecretTree.hasLeaf and Group.validateRemove compared the decoded value directly against the tree's leaf count, and a signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check. In the hasLeaf case the SenderData of an unprotected PrivateMessage could then drive LeafIndex.directPath through NodeIndex.parent() arithmetic that never reaches the tree root, growing the resulting node list without bound until the JVM exhausted its heap. A single small message from any current group member could therefore deny service to every other member of the group. Both comparisons now interpret the value as unsigned via Integer.toUnsignedLong, rejecting an out-of-range sender however it was encoded; well-formed leaf indices are unaffected.
Vulnerability data from NVD (NIST) · CISA KEV · EPSS