HiBench compatibility issues with Hadoop 3.x in micro.dfsioe prepare part... #546
Description
We met the error when preparing micro.dfsioe on CDH 6.0 where hadoop version is 3.0.
18/05/07 21:03:25 INFO dfsioe.TestDFSIOEnh: nrFiles = 2048 18/05/07 21:03:25 INFO dfsioe.TestDFSIOEnh: fileSize (MB) = 1000 18/05/07 21:03:25 INFO dfsioe.TestDFSIOEnh: bufferSize = 4096 Exception in thread "main" java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:760) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
So we rebuilded HiBench-7.0 using command
mvn -Phadoopbench -Psparkbench -Dhadoop.mr2.version=3.0.0 -Dspark=2.2 -Dscala=2.11 clean package,
it reports error
[ERROR] /root/HiBench_7.0/autogen/src/main/java/org/apache/hadoop/fs/dfsioe/TestDFSIO.java:[73,49] incompatible types: org.slf4j.Logger cannot be converted to org.apache.commons.logging.Log