Skip to content

mvn integration-test site fails enforce-ban-duplicate-classes #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
cburroughs opened this issue Jun 23, 2017 · 12 comments
Open

mvn integration-test site fails enforce-ban-duplicate-classes #270

cburroughs opened this issue Jun 23, 2017 · 12 comments
Assignees
@indianwhocodes
Labels
enhancement test-bug

Comments

@cburroughs
Copy link
Contributor 

mvn integration-test site
...<snip>...
[INFO] ------------------------------------------------------------------------
[INFO] Building java-manta-client-kryo-serialization 3.1.2-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-bytecode-version) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-circular-dependencies) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-duplicate-classes) @ java-manta-client-kryo-serialization ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    com.joyent.manta:java-manta-client:jar:3.1.2-SNAPSHOT:compile
    com.twmacinta:fast-md5:jar:2.7.1:compile
  Duplicate classes:
    com/twmacinta/util/MD5State.class
    com/twmacinta/util/MD5InputStream.class
    com/twmacinta/util/MD5.class

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] java-manta ......................................... SUCCESS [ 11.936 s]
[INFO] java-manta-client .................................. SUCCESS [ 40.718 s]
[INFO] java-manta-client-kryo-serialization ............... FAILURE [  0.254 s]
[INFO] java-manta-cli ..................................... SKIPPED
[INFO] java-manta-benchmark ............................... SKIPPED
[INFO] java-manta-it ...................................... SKIPPED
[INFO] java-manta-examples ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 53.091 s
[INFO] Finished at: 2017-06-23T17:23:41-04:00
[INFO] Final Memory: 81M/761M
```

I've tried to reduce this to a simpler set of goals to no avail.
@cburroughs
Copy link
Contributor Author

(I should also point out that I'm unsure if this is a "real" problem in the final jars or a strange corner case.)

@cburroughs
Copy link
Contributor Author

Possible related, I see the following warning when doing mvn -DskipTests=true package while shading the benchmark jar:

[WARNING] java-manta-client-3.1.3-SNAPSHOT.jar, fast-md5-2.7.1.jar define 3 overlapping classes: 
[WARNING]   - com.twmacinta.util.MD5State
[WARNING]   - com.twmacinta.util.MD5InputStream
[WARNING]   - com.twmacinta.util.MD5
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://maven.apache.org/plugins/maven-shade-plugin/

@cburroughs
Copy link
Contributor Author

In general we shade and relocate dependencies for the primary java-manta-client jar. So com.foo.Foo becomes com.joyent.manta.com.foo.Foo. This eliminates conflicts between the library versions java-manta uses, and what downstream applications are using.

The case of com.twmacinta.fast-md5 is complicated. We are shipping a class in the com.twmacinta package space to access package private classes. We can't relocate the package as that would relocate our own class, nor can we shade everything in that package but our class since the class would then loose the private access it needs.

Within the multi-module project I don't have a clear model for when a dependency on java-manta-client means: (A) give me those client classes (B) give me those client classes and their transitive dependencies (C) give me those client classes and their shaded transitive dependencies (D) give me those client classes, their their transitive dependencies, & their shaded transitive dependencies. I am also unsure if the answer changes based on the goal.

Adding a bunch of

                <exclusion>
                    <groupId>com.twmacinta</groupId>
                    <artifactId>fast-md5</artifactId>
                </exclusion>

to the intra-model dependencies appears to resolve all issues. I don't think that is harmful, but it is also not entirely satisfying.

@dekobon Do you have an historical context or advice on how to proceed?

@dekobon
Copy link
Contributor

dekobon commented Jun 28, 2017

I can't seem to reproduce this error. In fact, I can't seem to make the enforce plugin run at all now:

 ✘ elijah@Elijahs-MacBook-Pro  ~/Development/Joyent/java-manta   master ●  mvn enforcer:enforce
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] java-manta
[INFO] java-manta-client
[INFO] java-manta-client-kryo-serialization
[INFO] java-manta-cli
[INFO] java-manta-benchmark
[INFO] java-manta-it
[INFO] java-manta-examples
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building java-manta 3.1.3-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (default-cli) @ java-manta ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] java-manta ......................................... FAILURE [  0.519 s]
[INFO] java-manta-client .................................. SKIPPED
[INFO] java-manta-client-kryo-serialization ............... SKIPPED
[INFO] java-manta-cli ..................................... SKIPPED
[INFO] java-manta-benchmark ............................... SKIPPED
[INFO] java-manta-it ...................................... SKIPPED
[INFO] java-manta-examples ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.902 s
[INFO] Finished at: 2017-06-28T14:04:37-07:00
[INFO] Final Memory: 10M/245M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (default-cli) on project java-manta: The parameters 'rules' for goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce are missing or invalid -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginParameterException

@cburroughs
Copy link
Contributor Author

So ``mvn integration-test sitedoes not induce an error on your workstation? Does the behavior change if youclean` and `rm -r ~/.m2/repository/com/joyent/`?

My understanding is that we can't enforcer:enforce due to subtleties in how the pom is laid out: https://stackoverflow.com/a/6760365

@dekobon
Copy link
Contributor

dekobon commented Jun 30, 2017

I just tried again after doing rm -r ~/.m2/repository/com/joyent/. It still succeeded for me on my Mac. Can you reproduce this on a cloud instance? If so, I can ssh in and take a look.

@cburroughs
Copy link
Contributor Author

Cloud steps:

$ eval "$(triton env jpc-east-1)"
$ triton instance create ubuntu-16.04 g4-highcpu-1G -n mvndemo -w
Creating instance mvndemo (b521c6ac-c09d-6679-dfb3-ddd63b36e2bf, ubuntu-16.04@20170403)
Created instance mvndemo (b521c6ac-c09d-6679-dfb3-ddd63b36e2bf) in 18s
 $ triton ssh mvndemo
# apt-get update
# apt-get upgrade
# apt-get install openjdk-8-jre-headless openjdk-8-jdk-headless maven git
# git clone https://github.com/joyent/java-manta.git
# cd java-manta/
# mvn integration-test site
<MAVEN DOWNLOADS THE INTERNET>
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-maven) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-bytecode-version) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-circular-dependencies) @ java-manta-client-kryo-serialization ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-ban-duplicate-classes) @ java-manta-client-kryo-serialization ---
[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    com.joyent.manta:java-manta-client:jar:3.1.4-SNAPSHOT:compile
    com.twmacinta:fast-md5:jar:2.7.1:compile
  Duplicate classes:
    com/twmacinta/util/MD5State.class
    com/twmacinta/util/MD5InputStream.class
    com/twmacinta/util/MD5.class

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] java-manta ......................................... SUCCESS [ 52.365 s]
[INFO] java-manta-client .................................. SUCCESS [02:03 min]
[INFO] java-manta-client-kryo-serialization ............... FAILURE [  2.100 s]
[INFO] java-manta-cli ..................................... SKIPPED
[INFO] java-manta-benchmark ............................... SKIPPED
[INFO] java-manta-it ...................................... SKIPPED
[INFO] java-manta-examples ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:58 min
[INFO] Finished at: 2017-06-30T15:27:01+00:00
[INFO] Final Memory: 62M/162M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4.1:enforce (enforce-ban-duplicate-classes) on project java-manta-client-kryo-serialization: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :java-manta-client-kryo-serialization

Your keys listed on github should get you into 165.225.139.167

@dekobon
Copy link
Contributor

dekobon commented Jun 30, 2017 •
edited
Loading

Dumping the dependency tree for java-manta-client-kryo-serialization doesn't show a direct dependency on com.twmacinta:fast-md5:

$ mvn dependency:tree
[INFO] ------------------------------------------------------------------------
[INFO] Building java-manta-client-kryo-serialization 3.1.4-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:tree (default-cli) @ java-manta-client-kryo-serialization ---
[INFO] com.joyent.manta:java-manta-client-kryo-serialization:jar:3.1.4-SNAPSHOT
[INFO] +- com.joyent.manta:java-manta-client:jar:3.1.4-SNAPSHOT:compile
[INFO] |  +- org.bouncycastle:bcpkix-jdk15on:jar:1.56:compile
[INFO] |  +- org.bouncycastle:bcprov-jdk15on:jar:1.56:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.7.21:compile
[INFO] |  \- io.mikael:urlbuilder:jar:2.0.8:compile
[INFO] +- org.apache.commons:commons-lang3:jar:3.4:provided
[INFO] +- org.apache.commons:commons-collections4:jar:4.1:test
[INFO] +- commons-io:commons-io:jar:2.5:test
[INFO] +- com.esotericsoftware:kryo:jar:4.0.0:compile
[INFO] |  \- com.esotericsoftware:reflectasm:jar:1.11.3:compile
[INFO] |     \- org.ow2.asm:asm:jar:5.0.4:compile
[INFO] +- com.dorkbox:MinLog-SLF4J:jar:1.8:compile
[INFO] +- org.objenesis:objenesis:jar:2.2:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.1.7:test
[INFO] |  \- ch.qos.logback:logback-core:jar:1.1.7:test
[INFO] +- org.testng:testng:jar:6.9.10:test
[INFO] |  +- com.beust:jcommander:jar:1.48:test
[INFO] |  \- org.beanshell:bsh:jar:2.0b4:test
[INFO] +- org.slf4j:slf4j-api:jar:1.7.21:compile
[INFO] \- org.mockito:mockito-core:jar:2.0.2-beta:test
[INFO]    \- org.hamcrest:hamcrest-core:jar:1.1:test
...

I also inspected all of the pom.xml files and I can't find a direct reference to that dependency.

@dekobon
Copy link
Contributor

dekobon commented Jun 30, 2017 •
edited
Loading

I can keep the error from happening by doing:

<ignoreClasses>
    <ignoreClass>com.twmacinta.util.*</ignoreClass>
</ignoreClasses>

However, that doesn't address the root cause. It seems as though something is getting confused within the enforcer plugin. What is notable is that this is the only class that is being shaded but not relocated.

@dekobon
Copy link
Contributor

dekobon commented Jun 30, 2017

I tried mvn integration-test site and it failed. I tried mvn integration-test and it worked. I tried mvn site and that worked. I also tried mvn install site and that failed.I suspect that the combination of doing integration-test / install and site somehow confuses maven. I've seen this with other plugins before.

Many times when I'm setting up CI for Java, I will run different Maven goals in different stages because it helps to avoid this kind of weirdness. Most typically, I only run mvn site after mvn install has completed because a lot of the report plugins are not well maintained and they don't know how to manage unrealized dependency state.

I see two fixes: 1. Suppress the check for the specific class in question. I've already verified that it isn't duplicated in any of the final JARs. 2. Leave it as is and make a note that site should be run after running install or integration-test.

dekobon added a commit that referenced this issue Jul 12, 2017
@dekobon
Fixes #270. Added proper exclusions for shaded dependencies to module…
7ef748e 
… pom.xml files.
@indianwhocodes indianwhocodes self-assigned this Mar 29, 2019
@indianwhocodes
Copy link
Contributor

Latest Update on this issue since I am tracking it:

[WARNING] Rule 0: org.apache.maven.plugins.enforcer.BanDuplicateClasses failed with message:
Duplicate classes found:

  Found in:
    com.joyent.util:fast-md5:jar:2.7.1:compile
    com.joyent.manta:java-manta-client:jar:3.3.2-SNAPSHOT:compile
  Duplicate classes:
    com/twmacinta/io/NullOutputStream.class
    com/twmacinta/util/MD5InputStream.class
    com/twmacinta/util/MD5.class
    com/twmacinta/util/MD5State.class
    com/twmacinta/util/MD5OutputStream.class

  Found in:
    com.joyent.manta:java-manta-client:jar:3.3.2-SNAPSHOT:compile
    com.joyent.manta:java-manta-client-unshaded:jar:3.3.2-SNAPSHOT:compile

@indianwhocodes
Copy link
Contributor

PR #285 will have to be merged safely due to merge-conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@indianwhocodes indianwhocodes

Labels
enhancement test-bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@cburroughs @dekobon @indianwhocodes