8364655: Loading class with nested annotations causes stack overflow in VM

[jdksjolen]

Hi,

skip_annotation and skip_annotation_value are two mutually recursive functions calling each other in order to skip over classfile annotations. If a classfile contains a highly nested annotation, then this will lead to a stack overflow and a subsequent crash of the JVM. I propose that we insert a recursion limit to prevent this from happening.

This recursion limit will make the annotation parsing to bail out on the JVM side, skipping the rest of the annotations present. This is fine! The reason that the JVM parses these annotations is to gain access to a select few JDK-internal annotations, and we trust our own code to construct classfiles without egregious nesting. All classfile annotations are also parsed by Java-code, so we're not skipping general annotation parsing.

A regression test has been added.

Thanks!

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8364655: Loading class with nested annotations causes stack overflow in VM (Bug - P4)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/28674/head:pull/28674
$ git checkout pull/28674

Update a local copy of the PR:
$ git checkout pull/28674
$ git pull https://git.openjdk.org/jdk.git pull/28674/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 28674

View PR using the GUI difftool:
$ git pr show -t 28674

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/28674.diff

[bridgekeeper]

👋 Welcome back jsjolen! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

❗ This change is not yet ready to be integrated.
See the Progress checklist in the description for automated requirements.

[openjdk]

@jdksjolen The following label will be automatically applied to this pull request:

• hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.