那些在CDH5中是bug，到了CDH6版本就修复了的问题 hbaseHadoopCDHCloudera
程序员文章站
2022-03-11 17:19:54
...
盘点那些在CDH5中是bug，到了CDH6版本就修复了的问题。涉及到Hadoop、HDFS、YARN、HBASE、hive、hue、impala、kudu、oozie、solr、spark、kafka、parquet、zookeeper等组件。如果你的集群问题是被列出的这些，那么升级是可以解决问题的。
列出的只是部分的一百多个问题，Cloudera今年年底会停止CDH5的支持，对于CDH5的用户来说，升级是大势所趋。
问题 问题描述
HADOOP-12267 s3a failure due to integer overflow bug in AWS SDK
HADOOP-15169 "hadoop.ssl.enabled.protocols" should be considered in httpserver2
HADOOP-15812 ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console
HADOOP-15846 ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries.
HADOOP-15872 ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
HADOOP-15940 ABFS: For HNS account, avoid unnecessary get call when doing Rename
HADOOP-15948 Inconsistency in get and put syntax if filename/dirname contains space
HADOOP-15968 ABFS: getNamespaceEnabled can fail blocking user access thru ACLs
HADOOP-15969 ABFS: getNamespaceEnabled can fail blocking user access thru ACLs
HADOOP-15972 ABFS: reduce list page size to to 500
HADOOP-15975 ABFS: remove timeout check for DELETE and RENAME
HADOOP-16048 ABFS: Fix Date format parser
HADOOP-16461 Regression: FileSystem cache lock parses XML within the lock
HADOOP-16578 ABFS: fileSystemExists() should not call container level apis
HADOOP- 16587 OM and DN should persist SCM certificate as the trust root
HDFS-13193 Various Improvements for BlockTokenSecretManager
HDFS-13941 make storageId in BlockPoolTokenSecretManager.checkAccess optional
HDFS-14026 Overload BlockPoolTokenSecretManager.checkAccess to make storageId and storageType optional
HDFS-14366 Improve HDFS append performance
YARN- 9217 Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
YARN-9235 If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown
YARN-9337 GPU auto-discovery script runs even when the resource is given by hand
HBASE- 21991 Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
HBASE-22380 Break circle replication when doing bulkload
HBASE-23046 Remove compatibility case from truncate command
HIVE- 21999 Add sensitive ABFS configuration properties to HiveConf hidden list
HIVE-22236 Fail to create View selecting View containing NOT IN subquery
HUE-8946 [core] Add back name as argument to import LDAP group or user commands[useradmin] Fix argument as list in import_ldap_user and import_ldap_group
HUE-9011 Fix invalid delimiters in create Hive table
HUE-9019 Fix concurrent_user_session_limit failed after Django upgrade
HUE-9025 Fix multi query statement with invalidate metadata
HUE-9027 Fix erratic behaviour of the horizontal result scrollbar
IMPALA-6159 DataStreamSender should transparently handle some connection reset by peer
IMPALA-7802 Implement support for closing idle sessions
IMPALA-8333 Remove Impala Shell warnings part 2
IMPALA-8612 NPE when DropTableOrViewStmt analysis leaves serverName_ NULL
IMPALA-8673 Add query option to force plan hints for insert queries
IMPALA-8790 IllegalStateException: Illegal reference to non-materialized slot
IMPALA-8851 Drop table if exists throws authorization exception when table does not exist
IMPALA-8969 Grouping aggregator can cause segmentation fault when doing multiple aggregations.
KUDU-3014 Java client doesn't verify channel bindings during connection negotiation
KUDU-2980 Fault tolerant and diff scans fail if projection contains mis-ordered primary key columns
KUDU-2871 TLS 1.3 not supported by krpc
KUDU-2989 SASL server fails when FQDN is greater than 63 characters long
OOZIE-3464 Use UTF8 charset instead of default one
OOZIE-3543 Upgrade quartz to 2.3.1
SOLR-13532 Unable to start core recovery due to timeout in ping request
SOLR-13921 Processing UpdateRequest with delegation token throws NullPointerException
SENTRY-2535 SentryKafkaAuthorizer throws Exception when describing ACLs
SPARK-24621 WebUI - application 'name' urls point to http instead of https (even when ssl enabled)
SPARK-27453 DataFrameWriter.partitionBy is Silently Dropped by DSV1
SPARK-27621 Calling transform() method on a LinearRegressionModel throws NoSuchElementException
SPARK-29082 Spark driver cannot start with only delegation tokens
SPARK-29105 SHS may delete driver log file of in progress application
ZOOKEEPER-2251 testManyChildWatchersAutoReset is flaky
YARN-4212 airScheduler: Can't create a DRF queue under a FAIR policy queue
MAPREDUCE-6638 Do not attempt to recover progress from previous job attempts if spill encryption is enabled
YARN-1558 After apps are moved across queues, store new queue info in the RM state store
HBASE-7621 REST client (RemoteHTable) doesn't support binary row keys
Hive-11600 Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
HIVE-12727, refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
HIVE-15148 disallow loading data into bucketed tables (by default)
HIVE-18251 Loosen restriction for some checks
HIVE-18552 Split hive.strict.checks.large.query into two configs
HIVE-12609 Remove javaXML serialization
HIVE-15797 separate the configs for gby and oby position alias usage
HIVE-12442 HiveServer2: Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks
HIVE-12063 Pad Decimal numbers with trailing zeros to the scale of the column
HIVE-12237 Use slf4j as logging façade
HIVE-11304 Migrate to Log4j2 from Log4j 1.x
HIVE-6757 Remove deprecated parquet classes from outside of org.apache package
HIVE-12164 Remove jdbc stats collection mechanism
HIVE-12411 Remove counter based stats collection mechanism
HIVE-12005 Remove hbase based stats collection mechanism
HIVE-7575 GetTables thrift call is very slow
HIVE-11785 Support escaping carriage return and new line for LazySimpleSerDe
KAFKA-6252 A metric named 'XX' already exists, can't register another one.
KAFKA-5987 Kafka metrics templates used in document generation should maintain order of tags
KAFKA-5968 Remove all broker metrics during shutdown
KAFKA-5746 Add new metrics to support health checks
KAFKA-5738 Add cumulative count attribute for all Kafka rate metrics
KAFKA-5597 Autogenerate Producer sender metrics
KAFKA-5461 KIP-168: Add GlobalTopicCount metric per cluster
KAFKA-5341 Add UnderMinIsrPartitionCount and per-partition UnderMinIsr metrics
KAFKA-6258 SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data
KAFKA-5920 Handle SSL authentication failures as non-retriable exceptions in clients
KAFKA-5854 Handle SASL authentication failures as non-retriable exceptions in clients
KAFKA-5783 Implement KafkaPrincipalBuilder interface with support for SASL (KIP-189)
KAFKA-5720 In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with org.apache.kafka.common.errors.TimeoutException
KAFKA-5417 Clients get inconsistent connection states when SASL/SSL connection is marked CONECTED and DISCONNECTED at the same time
KAFKA-4764 Improve diagnostics for SASL authentication failures
KAFKA-6287 Inconsistent protocol type for empty consumer groups
KAFKA-5856 Add AdminClient.createPartitions()
KAFKA-5763 Refactor NetworkClient to use LogContext
KAFKA-5762 Refactor AdminClient to use LogContext
KAFKA-5755 Refactor Producer to use LogContext
KAFKA-5737 KafkaAdminClient thread should be daemon
KAFKA-5726 KafkaConsumer.subscribe() overload that takes just Pattern without ConsumerRebalanceListener
KAFKA-5629 Console Consumer overrides auto.offset.reset property when provided on the command line without warning about it.
KAFKA-5556 KafkaConsumer.commitSync throws IllegalStateException: Attempt to retrieve exception from future which hasn't failed
KAFKA-5534 KafkaConsumer offsetsForTimes should include partitions in result even if no offset could be found
KAFKA-5512 KafkaConsumer: High memory allocation rate when idle
KAFKA-4856 Calling KafkaProducer.close() from multiple threads may cause spurious error
KAFKA-4767 KafkaProducer is not joining its IO thread properly
KAFKA-4669 KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception
PARQUET-1217 Incorrect handling of missing values in Statistics
PARQUET-686 Allow for Unsigned Statistics in Binary Type
PARQUET-357 Parquet-thrift generates wrong schema for Thrift binary fields
PARQUET-753 GroupType.union() doesn't merge the original type
PARQUET-765 Upgrade Avro to 1.8.1
PARQUET-783 H2SeekableInputStream does not close its underlying FSDataInputStream, leading to connection leaks
PARQUET-791 Predicate pushing down on missing columns should work on UserDefinedPredicate too
PARQUET-806 Parquet-tools silently suppresses error messages
PARQUET-825 Static analyzer findings (NPEs, resource leaks)
PARQUET-1005 Fix DumpCommand parsing to allow column projection
PARQUET-1064 Deprecate type-defined sort ordering for INTERVAL type
PARQUET-1065 Deprecate type-defined sort ordering for INT96 type
PARQUET-1133 INT96 types and Maps without OriginalType cause exceptions in PigSchemaConverter
PARQUET-1141 IDs are dropped in metadata conversion
PARQUET-1152 Parquet-thrift doesn't compile with Thrift 0.9.3
PARQUET-1153 Parquet-thrift doesn't compile with Thrift 0.10.0
PARQUET-1185 TestBinary#testBinary unit test fails after PARQUET-1141
PARQUET-1191 Type.hashCode() takes originalType into account but Type.equals() does not
PARQUET-1208 Occasional endless loop in unit test
PARQUET-1217 Incorrect handling of missing values in Statistics
PARQUET-1246 Ignore float/double statistics in case of NaN
KUDU-2353 Add tooling to parse diagnostics log
KUDU-2290 Tool to re-create a tablet
KUDU-2399 Support IS NULL / IS NOT NULL predicates in Python
KUDU-2287 Add replica metric tracking time since there was a valid leader
KUDU-2427 Add support for Ubuntu 18.04
KUDU-1889 Support OpenSSL 1.1.0
KUDU-2012 Kudu Flume sink authn support
KUDU-2539 Supporting Spark Streaming DataFrame in KuduContext
KUDU-2529 kudu CLI command supports list the tablets under a table and list the replicas of a tablet
KUDU-16 Add server-side LIMIT for scanners
KUDU-1276 Add a vectorized read/write interface for pandas DataFrame objects
KUDU-2441 Unlike C++, Kudu Python API missing "set mutation buffer space"
KUDU-2095 Add scanner keepAlive method to the java client
KUDU-2563 Spark integration should use the scanner keep-alive API
KUDU-2368 Add ability to configure the number of reactors in KuduClient
KUDU-2395 Thread spike with all threads blocked in libnss
KUDU-2566 Enhance rowset tree pruning and discard string copy while querying
KUDU-1861 kudu test loadgen: change default behavior to avoid compactions on tablet servers
KUDU-2469 Handle CFile checksum failures
KUDU-2359 tserver should allow starting with a small number of missing data dirs
KUDU-2191 Hive Metastore Integration
KUDU-2242 Wait for NTP synchronization on startup
KUDU-2289 Tablet deletion should be throttled
ZOOKEEPER-2940 Deal with maxbuffer as it relates to large requests from clients
ZOOKEEPER-3019 Add a metric to track number of slow fsyncs
ZOOKEEPER-2994 Tool required to recover log and snapshot entries with CRC errors
列出的只是部分的一百多个问题，Cloudera今年年底会停止CDH5的支持，对于CDH5的用户来说，升级是大势所趋。
问题 问题描述
HADOOP-12267 s3a failure due to integer overflow bug in AWS SDK
HADOOP-15169 "hadoop.ssl.enabled.protocols" should be considered in httpserver2
HADOOP-15812 ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console
HADOOP-15846 ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries.
HADOOP-15872 ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2
HADOOP-15940 ABFS: For HNS account, avoid unnecessary get call when doing Rename
HADOOP-15948 Inconsistency in get and put syntax if filename/dirname contains space
HADOOP-15968 ABFS: getNamespaceEnabled can fail blocking user access thru ACLs
HADOOP-15969 ABFS: getNamespaceEnabled can fail blocking user access thru ACLs
HADOOP-15972 ABFS: reduce list page size to to 500
HADOOP-15975 ABFS: remove timeout check for DELETE and RENAME
HADOOP-16048 ABFS: Fix Date format parser
HADOOP-16461 Regression: FileSystem cache lock parses XML within the lock
HADOOP-16578 ABFS: fileSystemExists() should not call container level apis
HADOOP- 16587 OM and DN should persist SCM certificate as the trust root
HDFS-13193 Various Improvements for BlockTokenSecretManager
HDFS-13941 make storageId in BlockPoolTokenSecretManager.checkAccess optional
HDFS-14026 Overload BlockPoolTokenSecretManager.checkAccess to make storageId and storageType optional
HDFS-14366 Improve HDFS append performance
YARN- 9217 Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing
YARN-9235 If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown
YARN-9337 GPU auto-discovery script runs even when the resource is given by hand
HBASE- 21991 Fix MetaMetrics issues - [Race condition, Faulty remove logic], few improvements
HBASE-22380 Break circle replication when doing bulkload
HBASE-23046 Remove compatibility case from truncate command
HIVE- 21999 Add sensitive ABFS configuration properties to HiveConf hidden list
HIVE-22236 Fail to create View selecting View containing NOT IN subquery
HUE-8946 [core] Add back name as argument to import LDAP group or user commands[useradmin] Fix argument as list in import_ldap_user and import_ldap_group
HUE-9011 Fix invalid delimiters in create Hive table
HUE-9019 Fix concurrent_user_session_limit failed after Django upgrade
HUE-9025 Fix multi query statement with invalidate metadata
HUE-9027 Fix erratic behaviour of the horizontal result scrollbar
IMPALA-6159 DataStreamSender should transparently handle some connection reset by peer
IMPALA-7802 Implement support for closing idle sessions
IMPALA-8333 Remove Impala Shell warnings part 2
IMPALA-8612 NPE when DropTableOrViewStmt analysis leaves serverName_ NULL
IMPALA-8673 Add query option to force plan hints for insert queries
IMPALA-8790 IllegalStateException: Illegal reference to non-materialized slot
IMPALA-8851 Drop table if exists throws authorization exception when table does not exist
IMPALA-8969 Grouping aggregator can cause segmentation fault when doing multiple aggregations.
KUDU-3014 Java client doesn't verify channel bindings during connection negotiation
KUDU-2980 Fault tolerant and diff scans fail if projection contains mis-ordered primary key columns
KUDU-2871 TLS 1.3 not supported by krpc
KUDU-2989 SASL server fails when FQDN is greater than 63 characters long
OOZIE-3464 Use UTF8 charset instead of default one
OOZIE-3543 Upgrade quartz to 2.3.1
SOLR-13532 Unable to start core recovery due to timeout in ping request
SOLR-13921 Processing UpdateRequest with delegation token throws NullPointerException
SENTRY-2535 SentryKafkaAuthorizer throws Exception when describing ACLs
SPARK-24621 WebUI - application 'name' urls point to http instead of https (even when ssl enabled)
SPARK-27453 DataFrameWriter.partitionBy is Silently Dropped by DSV1
SPARK-27621 Calling transform() method on a LinearRegressionModel throws NoSuchElementException
SPARK-29082 Spark driver cannot start with only delegation tokens
SPARK-29105 SHS may delete driver log file of in progress application
ZOOKEEPER-2251 testManyChildWatchersAutoReset is flaky
YARN-4212 airScheduler: Can't create a DRF queue under a FAIR policy queue
MAPREDUCE-6638 Do not attempt to recover progress from previous job attempts if spill encryption is enabled
YARN-1558 After apps are moved across queues, store new queue info in the RM state store
HBASE-7621 REST client (RemoteHTable) doesn't support binary row keys
Hive-11600 Hive Parser to Support multi col in clause (x,y..) in ((..),..., ())
HIVE-12727, refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
HIVE-15148 disallow loading data into bucketed tables (by default)
HIVE-18251 Loosen restriction for some checks
HIVE-18552 Split hive.strict.checks.large.query into two configs
HIVE-12609 Remove javaXML serialization
HIVE-15797 separate the configs for gby and oby position alias usage
HIVE-12442 HiveServer2: Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks
HIVE-12063 Pad Decimal numbers with trailing zeros to the scale of the column
HIVE-12237 Use slf4j as logging façade
HIVE-11304 Migrate to Log4j2 from Log4j 1.x
HIVE-6757 Remove deprecated parquet classes from outside of org.apache package
HIVE-12164 Remove jdbc stats collection mechanism
HIVE-12411 Remove counter based stats collection mechanism
HIVE-12005 Remove hbase based stats collection mechanism
HIVE-7575 GetTables thrift call is very slow
HIVE-11785 Support escaping carriage return and new line for LazySimpleSerDe
KAFKA-6252 A metric named 'XX' already exists, can't register another one.
KAFKA-5987 Kafka metrics templates used in document generation should maintain order of tags
KAFKA-5968 Remove all broker metrics during shutdown
KAFKA-5746 Add new metrics to support health checks
KAFKA-5738 Add cumulative count attribute for all Kafka rate metrics
KAFKA-5597 Autogenerate Producer sender metrics
KAFKA-5461 KIP-168: Add GlobalTopicCount metric per cluster
KAFKA-5341 Add UnderMinIsrPartitionCount and per-partition UnderMinIsr metrics
KAFKA-6258 SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data
KAFKA-5920 Handle SSL authentication failures as non-retriable exceptions in clients
KAFKA-5854 Handle SASL authentication failures as non-retriable exceptions in clients
KAFKA-5783 Implement KafkaPrincipalBuilder interface with support for SASL (KIP-189)
KAFKA-5720 In Jenkins, kafka.api.SaslSslAdminClientIntegrationTest failed with org.apache.kafka.common.errors.TimeoutException
KAFKA-5417 Clients get inconsistent connection states when SASL/SSL connection is marked CONECTED and DISCONNECTED at the same time
KAFKA-4764 Improve diagnostics for SASL authentication failures
KAFKA-6287 Inconsistent protocol type for empty consumer groups
KAFKA-5856 Add AdminClient.createPartitions()
KAFKA-5763 Refactor NetworkClient to use LogContext
KAFKA-5762 Refactor AdminClient to use LogContext
KAFKA-5755 Refactor Producer to use LogContext
KAFKA-5737 KafkaAdminClient thread should be daemon
KAFKA-5726 KafkaConsumer.subscribe() overload that takes just Pattern without ConsumerRebalanceListener
KAFKA-5629 Console Consumer overrides auto.offset.reset property when provided on the command line without warning about it.
KAFKA-5556 KafkaConsumer.commitSync throws IllegalStateException: Attempt to retrieve exception from future which hasn't failed
KAFKA-5534 KafkaConsumer offsetsForTimes should include partitions in result even if no offset could be found
KAFKA-5512 KafkaConsumer: High memory allocation rate when idle
KAFKA-4856 Calling KafkaProducer.close() from multiple threads may cause spurious error
KAFKA-4767 KafkaProducer is not joining its IO thread properly
KAFKA-4669 KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception
PARQUET-1217 Incorrect handling of missing values in Statistics
PARQUET-686 Allow for Unsigned Statistics in Binary Type
PARQUET-357 Parquet-thrift generates wrong schema for Thrift binary fields
PARQUET-753 GroupType.union() doesn't merge the original type
PARQUET-765 Upgrade Avro to 1.8.1
PARQUET-783 H2SeekableInputStream does not close its underlying FSDataInputStream, leading to connection leaks
PARQUET-791 Predicate pushing down on missing columns should work on UserDefinedPredicate too
PARQUET-806 Parquet-tools silently suppresses error messages
PARQUET-825 Static analyzer findings (NPEs, resource leaks)
PARQUET-1005 Fix DumpCommand parsing to allow column projection
PARQUET-1064 Deprecate type-defined sort ordering for INTERVAL type
PARQUET-1065 Deprecate type-defined sort ordering for INT96 type
PARQUET-1133 INT96 types and Maps without OriginalType cause exceptions in PigSchemaConverter
PARQUET-1141 IDs are dropped in metadata conversion
PARQUET-1152 Parquet-thrift doesn't compile with Thrift 0.9.3
PARQUET-1153 Parquet-thrift doesn't compile with Thrift 0.10.0
PARQUET-1185 TestBinary#testBinary unit test fails after PARQUET-1141
PARQUET-1191 Type.hashCode() takes originalType into account but Type.equals() does not
PARQUET-1208 Occasional endless loop in unit test
PARQUET-1217 Incorrect handling of missing values in Statistics
PARQUET-1246 Ignore float/double statistics in case of NaN
KUDU-2353 Add tooling to parse diagnostics log
KUDU-2290 Tool to re-create a tablet
KUDU-2399 Support IS NULL / IS NOT NULL predicates in Python
KUDU-2287 Add replica metric tracking time since there was a valid leader
KUDU-2427 Add support for Ubuntu 18.04
KUDU-1889 Support OpenSSL 1.1.0
KUDU-2012 Kudu Flume sink authn support
KUDU-2539 Supporting Spark Streaming DataFrame in KuduContext
KUDU-2529 kudu CLI command supports list the tablets under a table and list the replicas of a tablet
KUDU-16 Add server-side LIMIT for scanners
KUDU-1276 Add a vectorized read/write interface for pandas DataFrame objects
KUDU-2441 Unlike C++, Kudu Python API missing "set mutation buffer space"
KUDU-2095 Add scanner keepAlive method to the java client
KUDU-2563 Spark integration should use the scanner keep-alive API
KUDU-2368 Add ability to configure the number of reactors in KuduClient
KUDU-2395 Thread spike with all threads blocked in libnss
KUDU-2566 Enhance rowset tree pruning and discard string copy while querying
KUDU-1861 kudu test loadgen: change default behavior to avoid compactions on tablet servers
KUDU-2469 Handle CFile checksum failures
KUDU-2359 tserver should allow starting with a small number of missing data dirs
KUDU-2191 Hive Metastore Integration
KUDU-2242 Wait for NTP synchronization on startup
KUDU-2289 Tablet deletion should be throttled
ZOOKEEPER-2940 Deal with maxbuffer as it relates to large requests from clients
ZOOKEEPER-3019 Add a metric to track number of slow fsyncs
ZOOKEEPER-2994 Tool required to recover log and snapshot entries with CRC errors