添加链接 注册    登录
link管理
链接快照平台
  • 输入网页链接，自动生成快照
  • 标签化管理网页链接
相关文章推荐
豪气的感冒药  ·  MySQL :: MySQL ...·  1 周前    · 
气宇轩昂的香瓜  ·  Copy and transform ...·  1 周前    · 
干练的麻辣香锅  ·  使用 DataAdapter 更新数据源 ...·  4 天前    · 
逃跑的企鹅  ·  DataAdapter DataTable ...·  4 天前    · 
行走的苹果  ·  使用 Oracle ...·  昨天    · 
近视的针织衫  ·  关于规范网络转载版权秩序的通知-其他行政性规 ...·  7 月前    · 
逆袭的甘蔗  ·  Journal of Medical ...·  9 月前    · 
机灵的乌冬面  ·  西游记最厉害的兵器是什么？不是九齿耙，不是金 ...·  10 月前    · 
大气的香槟  ·  如何正确地使用Matlab编码器进行动态和静 ...·  11 月前    · 
考研的围巾  ·  docker一：容器介绍与lxc容器 - ...·  1 年前    · 
link管理  ›  2019 Changelog | ClickHouse Docs
table
https://clickhouse.com/docs/en/whats-new/changelog/2019
奋斗的香蕉
1 年前
Skip to main content

2019 Changelog

ClickHouse Release 19.17 ​

ClickHouse Release 19.17.6.36, 2019-12-27 ​

Bug Fix ​

  • Fixed potential buffer overflow in decompress. Malicious user can pass fabricated compressed data that could cause read after buffer. This issue was found by Eldar Zaitov from Yandex information security team. #8404 ( alexey-milovidov )
  • Fixed possible server crash ( std::terminate ) when the server cannot send or write data in JSON or XML format with values of String data type (that require UTF-8 validation) or when compressing result data with Brotli algorithm or in some other rare cases. #8384 ( alexey-milovidov )
  • Fixed dictionaries with source from a clickhouse VIEW , now reading such dictionaries does not cause the error There is no query . #8351 ( Nikolai Kochetov )
  • Fixed checking if a client host is allowed by host_regexp specified in users.xml. #8241 , #8342 ( Vitaly Baranov )
  • RENAME TABLE for a distributed table now renames the folder containing inserted data before sending to shards. This fixes an issue with successive renames tableA->tableB , tableC->tableA . #8306 ( tavplubix )
  • range_hashed external dictionaries created by DDL queries now allow ranges of arbitrary numeric types. #8275 ( alesapin )
  • Fixed INSERT INTO table SELECT ... FROM mysql(...) table function. #8234 ( tavplubix )
  • Fixed segfault in INSERT INTO TABLE FUNCTION file() while inserting into a file which does not exist. Now in this case file would be created and then insert would be processed. #8177 ( Olga Khvostikova )
  • Fixed bitmapAnd error when intersecting an aggregated bitmap and a scalar bitmap. #8082 ( Yue Huang )
  • Fixed segfault when EXISTS query was used without TABLE or DICTIONARY qualifier, just like EXISTS t . #8213 ( alexey-milovidov )
  • Fixed return type for functions rand and randConstant in case of nullable argument. Now functions always return UInt32 and never Nullable(UInt32) . #8204 ( Nikolai Kochetov )
  • Fixed DROP DICTIONARY IF EXISTS db.dict , now it does not throw exception if db does not exist. #8185 ( Vitaly Baranov )
  • If a table wasn’t completely dropped because of server crash, the server will try to restore and load it #8176 ( tavplubix )
  • Fixed a trivial count query for a distributed table if there are more than two shard local table. #8164 ( 小路 )
  • Fixed bug that lead to a data race in DB::BlockStreamProfileInfo::calculateRowsBeforeLimit() #8143 ( Alexander Kazakov )
  • Fixed ALTER table MOVE part executed immediately after merging the specified part, which could cause moving a part which the specified part merged into. Now it correctly moves the specified part. #8104 ( Vladimir Chebotarev )
  • Expressions for dictionaries can be specified as strings now. This is useful for calculation of attributes while extracting data from non-ClickHouse sources because it allows to use non-ClickHouse syntax for those expressions. #8098 ( alesapin )
  • Fixed a very rare race in clickhouse-copier because of an overflow in ZXid. #8088 ( Ding Xiang Fei )
  • Fixed the bug when after the query failed (due to “Too many simultaneous queries” for example) it would not read external tables info, and the next request would interpret this info as the beginning of the next query causing an error like Unknown packet from client . #8084 ( Azat Khuzhin )
  • Avoid null dereference after “Unknown packet X from server” #8071 ( Azat Khuzhin )
  • Restore support of all ICU locales, add the ability to apply collations for constant expressions and add language name to system.collations table. #8051 ( alesapin )
  • Number of streams for read from StorageFile and StorageHDFS is now limited, to avoid exceeding the memory limit. #7981 ( alesapin )
  • Fixed CHECK TABLE query for *MergeTree tables without key. #7979 ( alesapin )
  • Removed the mutation number from a part name in case there were no mutations. This removing improved the compatibility with older versions. #8250 ( alesapin )
  • Fixed the bug that mutations are skipped for some attached parts due to their data_version are larger than the table mutation version. #7812 ( Zhichang Yu )
  • Allow starting the server with redundant copies of parts after moving them to another device. #7810 ( Vladimir Chebotarev )
  • Fixed the error “Sizes of columns does not match” that might appear when using aggregate function columns. #7790 ( Boris Granveaud )
  • Now an exception will be thrown in case of using WITH TIES alongside LIMIT BY. And now it’s possible to use TOP with LIMIT BY. #7637 ( Nikita Mikhaylov )
  • Fix dictionary reload if it has invalidate_query , which stopped updates and some exception on previous update tries. #8029 ( alesapin )

ClickHouse Release 19.17.4.11, 2019-11-22 ​

Backward Incompatible Change ​

  • Using column instead of AST to store scalar subquery results for better performance. Setting enable_scalar_subquery_optimization was added in 19.17 and it was enabled by default. It leads to errors like this during upgrade to 19.17.2 or 19.17.3 from previous versions. This setting was disabled by default in 19.17.4, to make possible upgrading from 19.16 and older versions without errors. #7392 ( Amos Bird )

New Feature ​

Experimental Feature ​

Bug Fix ​

Improvement ​

Performance Improvement ​

Build/Testing/Packaging Improvement ​

Other ​

ClickHouse Release 19.16 ​

ClickHouse Release 19.16.14.65, 2020-03-25 ​

  • Fixed up a bug in batched calculations of ternary logical OPs on multiple arguments (more than 10). #8718 ( Alexander Kazakov ) This bugfix was backported to version 19.16 by a special request from Altinity.

ClickHouse Release 19.16.14.65, 2020-03-05 ​

  • Fix distributed subqueries incompatibility with older CH versions. Fixes #7851 (tabplubix)
  • When executing CREATE query, fold constant expressions in storage engine arguments. Replace empty database name with current database. Fixes #6508 , #3492 . Also fix check for local address in ClickHouseDictionarySource . #9262 (tabplubix)
  • Now background merges in *MergeTree table engines family preserve storage policy volume order more accurately. #8549 ( Vladimir Chebotarev )
  • Prevent losing data in Kafka in rare cases when exception happens after reading suffix but before commit. Fixes #9378 . Related: #7175 #9507 (filimonov)
  • Fix bug leading to server termination when trying to use / drop Kafka table created with wrong parameters. Fixes #9494 . Incorporates #9507 . #9513 (filimonov)
  • Allow using MaterializedView with subqueries above Kafka tables. #8197 ( filimonov )

New Feature ​

  • Add deduplicate_blocks_in_dependent_materialized_views option to control the behaviour of idempotent inserts into tables with materialized views. This new feature was added to the bugfix release by a special request from Altinity. #9070 (urykhy)

ClickHouse Release 19.16.2.2, 2019-10-30 ​

Backward Incompatible Change ​

New Feature ​

  • Introduce uniqCombined64() to calculate cardinality greater than UINT_MAX. #7213 , #7222 ( Azat Khuzhin )
  • Support Bloom filter indexes on Array columns. #6984 ( achimbab )
  • Add a function getMacro(name) that returns String with the value of corresponding <macros> from server configuration. #7240 ( alexey-milovidov )
  • Set two configuration options for a dictionary based on an HTTP source: credentials and http-headers . #7092 ( Guillaume Tassery )
  • Add a new ProfileEvent Merge that counts the number of launched background merges. #7093 ( Mikhail Korotov )
  • Add fullHostName function that returns a fully qualified domain name. #7263 #7291 ( sundyli )
  • Add function arraySplit and arrayReverseSplit which split an array by “cut off” conditions. They are useful in time sequence handling. #7294 ( hcz )
  • Add new functions that return the Array of all matched indices in multiMatch family of functions. #7299 ( Danila Kutenin )
  • Add a new database engine Lazy that is optimized for storing a large number of small -Log tables. #7171 ( Nikita Vasilev )
  • Add aggregate functions groupBitmapAnd, -Or, -Xor for bitmap columns. #7109 ( Zhichang Yu )
  • Add aggregate function combinators -OrNull and -OrDefault, which return null or default values when there is nothing to aggregate. #7331 ( hcz )
  • Introduce CustomSeparated data format that supports custom escaping and delimiter rules. #7118 ( tavplubix )
  • Support Redis as source of external dictionary. #4361 #6962 ( comunodi , Anton Popov )

Bug Fix ​

  • Fix wrong query result if it has WHERE IN (SELECT ...) section and optimize_read_in_order is used. #7371 ( Anton Popov )
  • Disabled MariaDB authentication plugin, which depends on files outside of project. #7140 ( Yuriy Baranov )
  • Fix exception Cannot convert column ... because it is constant but values of constants are different in source and result which could rarely happen when functions now() , today() , yesterday() , randConstant() are used. #7156 ( Nikolai Kochetov )
  • Fixed issue of using HTTP keep alive timeout instead of TCP keep alive timeout. #7351 ( Vasily Nemkov )
  • Fixed a segmentation fault in groupBitmapOr (issue #7109 ). #7289 ( Zhichang Yu )
  • For materialized views the commit for Kafka is called after all data were written. #7175 ( Ivan )
  • Fixed wrong duration_ms value in system.part_log table. It was ten times off. #7172 ( Vladimir Chebotarev )
  • A quick fix to resolve crash in LIVE VIEW table and re-enabling all LIVE VIEW tests. #7201 ( vzakaznikov )
  • Serialize NULL values correctly in min/max indexes of MergeTree parts. #7234 ( Alexander Kuzmenkov )
  • Don’t put virtual columns to .sql metadata when table is created as CREATE TABLE AS . #7183 ( Ivan )
  • Fix segmentation fault in ATTACH PART query. #7185 ( alesapin )
  • Fix wrong result for some queries given by the optimization of empty IN subqueries and empty INNER/RIGHT JOIN. #7284 ( Nikolai Kochetov )
  • Fixing AddressSanitizer error in the LIVE VIEW getHeader() method. #7271 ( vzakaznikov )

Improvement ​

Performance Improvement ​

Build/Testing/Packaging Improvement ​

Code Cleanup ​

ClickHouse Release 19.15 ​

ClickHouse Release 19.15.4.10, 2019-10-31 ​

Bug Fix ​

ClickHouse Release 19.15.3.6, 2019-10-09 ​

Bug Fix ​

ClickHouse Release 19.15.2.2, 2019-10-01 ​

New Feature ​

Experimental Feature ​

  • Implement (in memory) Merge Join variant that does not change current pipeline. Result is partially sorted by merge key. Set partial_merge_join = 1 to use this feature. The Merge Join is still in development. #6940 ( Artem Zuikov )
  • Add S3 engine and table function. It is still in development (no authentication support yet). #5596 ( Vladimir Chebotarev )

Improvement ​

  • Every message read from Kafka is inserted atomically. This resolves almost all known issues with Kafka engine. #6950 ( Ivan )
  • Improvements for failover of Distributed queries. Shorten recovery time, also it is now configurable and can be seen in system.clusters . #6399 ( Vasily Nemkov )
  • Support numeric values for Enums directly in IN section. #6766 #6941 ( dimarub2000 )
  • Support (optional, disabled by default) redirects on URL storage. #6914 ( maqroll )
  • Add information message when client with an older version connects to a server. #6893 ( Philipp Malkovsky )
  • Remove maximum backoff sleep time limit for sending data in Distributed tables #6895 ( Azat Khuzhin )
  • Add ability to send profile events (counters) with cumulative values to graphite. It can be enabled under <events_cumulative> in server config.xml . #6969 ( Azat Khuzhin )
  • Add automatically cast type T to LowCardinality(T) while inserting data in column of type LowCardinality(T) in Native format via HTTP. #6891 ( Nikolai Kochetov )
  • Add ability to use function hex without using reinterpretAsString for Float32 , Float64 . #7024 ( Mikhail Korotov )

Build/Testing/Packaging Improvement ​

Bug Fix ​

Backward Incompatible Change ​

  • Changed serialization format of bitmap * aggregate function states to improve performance. Serialized states of bitmap * from previous versions cannot be read. #6908 ( Zhichang Yu )

ClickHouse Release 19.14 ​

ClickHouse Release 19.14.7.15, 2019-10-02 ​

Bug Fix ​

  • This release also contains all bug fixes from 19.11.12.69.
  • Fixed compatibility for distributed queries between 19.14 and earlier versions. This fixes #7068 . #7069 ( alexey-milovidov )

ClickHouse Release 19.14.6.12, 2019-09-19 ​

Bug Fix ​

  • Fix for function АrrayEnumerateUniqRanked with empty arrays in params. #6928 ( proller )
  • Fixed subquery name in queries with ARRAY JOIN and GLOBAL IN subquery with alias. Use subquery alias for external table name if it is specified. #6934 ( Ivan )

Build/Testing/Packaging Improvement ​

  • Fix flapping test 00715_fetch_merged_or_mutated_part_zookeeper by rewriting it to a shell scripts because it needs to wait for mutations to apply. #6977 ( Alexander Kazakov )
  • Fixed UBSan and MemSan failure in function groupUniqArray with emtpy array argument. It was caused by placing of empty PaddedPODArray into hash table zero cell because constructor for zero cell value was not called. #6937 ( Amos Bird )

ClickHouse Release 19.14.3.3, 2019-09-10 ​

New Feature ​

  • WITH FILL modifier for ORDER BY . (continuation of #5069 ) #6610 ( Anton Popov )
  • WITH TIES modifier for LIMIT . (continuation of #5069 ) #6610 ( Anton Popov )
  • Parse unquoted NULL literal as NULL (if setting format_csv_unquoted_null_literal_as_null=1 ). Initialize null fields with default values if data type of this field is not nullable (if setting input_format_null_as_default=1 ). #5990 #6055 ( tavplubix )
  • Support for wildcards in paths of table functions file and hdfs . If the path contains wildcards, the table will be readonly. Example of usage: select * from hdfs('hdfs://hdfs1:9000/some_dir/another_dir/*/file{0..9}{0..9}') and select * from file('some_dir/{some_file,another_file,yet_another}.tsv', 'TSV', 'value UInt32') . #6092 ( Olga Khvostikova )
  • New system.metric_log table which stores values of system.events and system.metrics with specified time interval. #6363 #6467 ( Nikita Mikhaylov ) #6530 ( alexey-milovidov )
  • Allow to write ClickHouse text logs to system.text_log table. #6037 #6103 ( Nikita Mikhaylov ) #6164 ( alexey-milovidov )
  • Show private symbols in stack traces (this is done via parsing symbol tables of ELF files). Added information about file and line number in stack traces if debug info is present. Speedup symbol name lookup with indexing symbols present in program. Added new SQL functions for introspection: demangle and addressToLine . Renamed function symbolizeAddress to addressToSymbol for consistency. Function addressToSymbol will return mangled name for performance reasons and you have to apply demangle . Added setting allow_introspection_functions which is turned off by default. #6201 ( alexey-milovidov )
  • Table function values (the name is case-insensitive). It allows to read from VALUES list proposed in #5984 . Example: SELECT * FROM VALUES('a UInt64, s String', (1, 'one'), (2, 'two'), (3, 'three')) . #6217 . #6209 ( dimarub2000 )
  • Added an ability to alter storage settings. Syntax: ALTER TABLE <table> MODIFY SETTING <setting> = <value> . #6366 #6669 #6685 ( alesapin )
  • Support for removing of detached parts. Syntax: ALTER TABLE <table_name> DROP DETACHED PART '<part_id>' . #6158 ( tavplubix )
  • Table constraints. Allows to add constraint to table definition which will be checked at insert. #5273 ( Gleb Novikov ) #6652 ( alexey-milovidov )
  • Suppport for cascaded materialized views. #6324 ( Amos Bird )
  • Turn on query profiler by default to sample every query execution thread once a second. #6283 ( alexey-milovidov )
  • Input format ORC . #6454 #6703 ( akonyaev90 )
  • Added two new functions: sigmoid and tanh (that are useful for machine learning applications). #6254 ( alexey-milovidov )
  • Function hasToken(haystack, token) , hasTokenCaseInsensitive(haystack, token) to check if given token is in haystack. Token is a maximal length substring between two non alphanumeric ASCII characters (or boundaries of haystack). Token must be a constant string. Supported by tokenbf_v1 index specialization. #6596 , #6662 ( Vasily Nemkov )
  • New function neighbor(value, offset[, default_value]) . Allows to reach prev/next value within column in a block of data. #5925 ( Alex Krash ) 6685365ab8c5b74f9650492c88a012596eb1b0c6 341e2e4587a18065c2da1ca888c73389f48ce36c Alexey Milovidov
  • Created a function currentUser() , returning login of authorized user. Added alias user() for compatibility with MySQL. #6470 ( Alex Krash )
  • New aggregate functions quantilesExactInclusive and quantilesExactExclusive which were proposed in #5885 . #6477 ( dimarub2000 )
  • Function bitmapRange(bitmap, range_begin, range_end) which returns new set with specified range (not include the range_end ). #6314 ( Zhichang Yu )
  • Function geohashesInBox(longitude_min, latitude_min, longitude_max, latitude_max, precision) which creates array of precision-long strings of geohash-boxes covering provided area. #6127 ( Vasily Nemkov )
  • Implement support for INSERT query with Kafka tables. #6012 ( Ivan )
  • Added support for _partition and _timestamp virtual columns to Kafka engine. #6400 ( Ivan )
  • Possibility to remove sensitive data from query_log , server logs, process list with regexp-based rules. #5710 ( filimonov )

Experimental Feature ​

Bug Fix ​

  • This release also contains all bug fixes from 19.13 and 19.11.
  • Fix segmentation fault when the table has skip indices and vertical merge happens. #6723 ( alesapin )
  • Fix per-column TTL with non-trivial column defaults. Previously in case of force TTL merge with OPTIMIZE ... FINAL query, expired values was replaced by type defaults instead of user-specified column defaults. #6796 ( Anton Popov )
  • Fix Kafka messages duplication problem on normal server restart. #6597 ( Ivan )
  • Fixed infinite loop when reading Kafka messages. Do not pause/resume consumer on subscription at all - otherwise it may get paused indefinitely in some scenarios. #6354 ( Ivan )
  • Fix Key expression contains comparison between inconvertible types exception in bitmapContains function. #6136 #6146 #6156 ( dimarub2000 )
  • Fix segfault with enabled optimize_skip_unused_shards and missing sharding key. #6384 ( Anton Popov )
  • Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address 0x14c0 that may happed due to concurrent DROP TABLE and SELECT from system.parts or system.parts_columns . Fixed race condition in preparation of mutation queries. Fixed deadlock caused by OPTIMIZE of Replicated tables and concurrent modification operations like ALTERs. #6514 ( alexey-milovidov )
  • Removed extra verbose logging in MySQL interface #6389 ( alexey-milovidov )
  • Return the ability to parse boolean settings from ‘true’ and ‘false’ in the configuration file. #6278 ( alesapin )
  • Fix crash in quantile and median function over Nullable(Decimal128) . #6378 ( Artem Zuikov )
  • Fixed possible incomplete result returned by SELECT query with WHERE condition on primary key contained conversion to Float type. It was caused by incorrect checking of monotonicity in toFloat function. #6248 #6374 ( dimarub2000 )
  • Check max_expanded_ast_elements setting for mutations. Clear mutations after TRUNCATE TABLE . #6205 ( Winter Zhang )
  • Fix JOIN results for key columns when used with join_use_nulls . Attach Nulls instead of columns defaults. #6249 ( Artem Zuikov )
  • Fix for skip indices with vertical merge and alter. Fix for Bad size of marks file exception. #6594 #6713 ( alesapin )
  • Fix rare crash in ALTER MODIFY COLUMN and vertical merge when one of merged/altered parts is empty (0 rows) #6746 #6780 ( alesapin )
  • Fixed bug in conversion of LowCardinality types in AggregateFunctionFactory . This fixes #6257 . #6281 ( Nikolai Kochetov )
  • Fix wrong behavior and possible segfaults in topK and topKWeighted aggregated functions. #6404 ( Anton Popov )
  • Fixed unsafe code around getIdentifier function. #6401 #6409 ( alexey-milovidov )
  • Fixed bug in MySQL wire protocol (is used while connecting to ClickHouse form MySQL client). Caused by heap buffer overflow in PacketPayloadWriteBuffer . #6212 ( Yuriy Baranov )
  • Fixed memory leak in bitmapSubsetInRange function. #6819 ( Zhichang Yu )
  • Fix rare bug when mutation executed after granularity change. #6816 ( alesapin )
  • Allow protobuf message with all fields by default. #6132 ( Vitaly Baranov )
  • Resolve a bug with nullIf function when we send a NULL argument on the second argument. #6446 ( Guillaume Tassery )
  • Fix rare bug with wrong memory allocation/deallocation in complex key cache dictionaries with string fields which leads to infinite memory consumption (looks like memory leak). Bug reproduces when string size was a power of two starting from eight (8, 16, 32, etc). #6447 ( alesapin )
  • Fixed Gorilla encoding on small sequences which caused exception Cannot write after end of buffer . #6398 #6444 ( Vasily Nemkov )
  • Allow to use not nullable types in JOINs with join_use_nulls enabled. #6705 ( Artem Zuikov )
  • Disable Poco::AbstractConfiguration substitutions in query in clickhouse-client . #6706 ( alexey-milovidov )
  • Avoid deadlock in REPLACE PARTITION . #6677 ( alexey-milovidov )
  • Using arrayReduce for constant arguments may lead to segfault. #6242 #6326 ( alexey-milovidov )
  • Fix inconsistent parts which can appear if replica was restored after DROP PARTITION . #6522 #6523 ( tavplubix )
  • Fixed hang in JSONExtractRaw function. #6195 #6198 ( alexey-milovidov )
  • Fix bug with incorrect skip indices serialization and aggregation with adaptive granularity. #6594 . #6748 ( alesapin )
  • Fix WITH ROLLUP and WITH CUBE modifiers of GROUP BY with two-level aggregation. #6225 ( Anton Popov )
  • Fix bug with writing secondary indices marks with adaptive granularity. #6126 ( alesapin )
  • Fix initialization order while server startup. Since StorageMergeTree::background_task_handle is initialized in startup() the MergeTreeBlockOutputStream::write() may try to use it before initialization. Just check if it is initialized. #6080 ( Ivan )
  • Clearing the data buffer from the previous read operation that was completed with an error. #6026 ( Nikolay )
  • Fix bug with enabling adaptive granularity when creating a new replica for Replicated * MergeTree table. #6394 #6452 ( alesapin )
  • Fixed possible crash during server startup in case of exception happened in libunwind during exception at access to uninitialized ThreadStatus structure. #6456 ( Nikita Mikhaylov )
  • Fix crash in yandexConsistentHash function. Found by fuzz test. #6304 #6305 ( alexey-milovidov )
  • Fixed the possibility of hanging queries when server is overloaded and global thread pool becomes near full. This have higher chance to happen on clusters with large number of shards (hundreds), because distributed queries allocate a thread per connection to each shard. For example, this issue may reproduce if a cluster of 330 shards is processing 30 concurrent distributed queries. This issue affects all versions starting from 19.2. #6301 ( alexey-milovidov )
  • Fixed logic of arrayEnumerateUniqRanked function. #6423 ( alexey-milovidov )
  • Fix segfault when decoding symbol table. #6603 ( Amos Bird )
  • Fixed irrelevant exception in cast of LowCardinality(Nullable) to not-Nullable column in case if it does not contain Nulls (e.g. in query like SELECT CAST(CAST('Hello' AS LowCardinality(Nullable(String))) AS String) . #6094 #6119 ( Nikolai Kochetov )
  • Removed extra quoting of description in system.settings table. #6696 #6699 ( alexey-milovidov )
  • Avoid possible deadlock in TRUNCATE of Replicated table. #6695 ( alexey-milovidov )
  • Fix reading in order of sorting key. #6189 ( Anton Popov )
  • Fix ALTER TABLE ... UPDATE query for tables with enable_mixed_granularity_parts=1 . #6543 ( alesapin )
  • Fix bug opened by #4405 (since 19.4.0). Reproduces in queries to Distributed tables over MergeTree tables when we does not query any columns ( SELECT 1 ). #6236 ( alesapin )
  • Fixed overflow in integer division of signed type to unsigned type. The behaviour was exactly as in C or C++ language (integer promotion rules) that may be surprising. Please note that the overflow is still possible when dividing large signed number to large unsigned number or vice-versa (but that case is less usual). The issue existed in all server versions. #6214 #6233 ( alexey-milovidov )
  • Limit maximum sleep time for throttling when max_execution_speed or max_execution_speed_bytes is set. Fixed false errors like Estimated query execution time (inf seconds) is too long . #5547 #6232 ( alexey-milovidov )
  • Fixed issues about using MATERIALIZED columns and aliases in MaterializedView . #448 #3484 #3450 #2878 #2285 #3796 ( Amos Bird ) #6316 ( alexey-milovidov )
  • Fix FormatFactory behaviour for input streams which are not implemented as processor. #6495 ( Nikolai Kochetov )
  • Fixed typo. #6631 ( Alex Ryndin )
  • Typo in the error message ( is - > are ). #6839 ( Denis Zhuravlev )
  • Fixed error while parsing of columns list from string if type contained a comma (this issue was relevant for File , URL , HDFS storages) #6217 . #6209 ( dimarub2000 )

Security Fix ​

  • This release also contains all bug security fixes from 19.13 and 19.11.
  • Fixed the possibility of a fabricated query to cause server crash due to stack overflow in SQL parser. Fixed the possibility of stack overflow in Merge and Distributed tables, materialized views and conditions for row-level security that involve subqueries. #6433 ( alexey-milovidov )

Improvement ​

  • Correct implementation of ternary logic for AND/OR . #6048 ( Alexander Kazakov )
  • Now values and rows with expired TTL will be removed after OPTIMIZE ... FINAL query from old parts without TTL infos or with outdated TTL infos, e.g. after ALTER ... MODIFY TTL query. Added queries SYSTEM STOP/START TTL MERGES to disallow/allow assign merges with TTL and filter expired values in all merges. #6274 ( Anton Popov )
  • Possibility to change the location of ClickHouse history file for client using CLICKHOUSE_HISTORY_FILE env. #6840 ( filimonov )
  • Remove dry_run flag from InterpreterSelectQuery . … #6375 ( Nikolai Kochetov )
  • Support ASOF JOIN with ON section. #6211 ( Artem Zuikov )
  • Better support of skip indexes for mutations and replication. Support for MATERIALIZE/CLEAR INDEX ... IN PARTITION query. UPDATE x = x recalculates all indices that use column x . #5053 ( Nikita Vasilev )
  • Allow to ATTACH live views (for example, at the server startup) regardless to allow_experimental_live_view setting. #6754 ( alexey-milovidov )
  • For stack traces gathered by query profiler, do not include stack frames generated by the query profiler itself. #6250 ( alexey-milovidov )
  • Now table functions values , file , url , hdfs have support for ALIAS columns. #6255 ( alexey-milovidov )
  • Throw an exception if config.d file does not have the corresponding root element as the config file. #6123 ( dimarub2000 )
  • Print extra info in exception message for no space left on device . #6182 , #6252 #6352 ( tavplubix )
  • When determining shards of a Distributed table to be covered by a read query (for optimize_skip_unused_shards = 1) ClickHouse now checks conditions from both prewhere and where clauses of select statement. #6521 ( Alexander Kazakov )
  • Enabled SIMDJSON for machines without AVX2 but with SSE 4.2 and PCLMUL instruction set. #6285 #6320 ( alexey-milovidov )
  • ClickHouse can work on filesystems without O_DIRECT support (such as ZFS and BtrFS) without additional tuning. #4449 #6730 ( alexey-milovidov )
  • Support push down predicate for final subquery. #6120 ( TCeason ) #6162 ( alexey-milovidov )
  • Better JOIN ON keys extraction #6131 ( Artem Zuikov )
  • Upated SIMDJSON . #6285 . #6306 ( alexey-milovidov )
  • Optimize selecting of smallest column for SELECT count() query. #6344 ( Amos Bird )
  • Added strict parameter in windowFunnel() . When the strict is set, the windowFunnel() applies conditions only for the unique values. #6548 ( achimbab )
  • Safer interface of mysqlxx::Pool . #6150 ( avasiliev )
  • Options line size when executing with --help option now corresponds with terminal size. #6590 ( dimarub2000 )
  • Disable “read in order” optimization for aggregation without keys. #6599 ( Anton Popov )
  • HTTP status code for INCORRECT_DATA and TYPE_MISMATCH error codes was changed from default 500 Internal Server Error to 400 Bad Request . #6271 ( Alexander Rodin )
  • Move Join object from ExpressionAction into AnalyzedJoin . ExpressionAnalyzer and ExpressionAction do not know about Join class anymore. Its logic is hidden by AnalyzedJoin iface. #6801 ( Artem Zuikov )
  • Fixed possible deadlock of distributed queries when one of shards is localhost but the query is sent via network connection. #6759 ( alexey-milovidov )
  • Changed semantic of multiple tables RENAME to avoid possible deadlocks. #6757 . #6756 ( alexey-milovidov )
  • Rewritten MySQL compatibility server to prevent loading full packet payload in memory. Decreased memory consumption for each connection to approximately 2 * DBMS_DEFAULT_BUFFER_SIZE (read/write buffers). #5811 ( Yuriy Baranov )
  • Move AST alias interpreting logic out of parser that does not have to know anything about query semantics. #6108 ( Artem Zuikov )
  • Slightly more safe parsing of NamesAndTypesList . #6408 . #6410 ( alexey-milovidov )
  • clickhouse-copier : Allow use where_condition from config with partition_key alias in query for checking partition existence (Earlier it was used only in reading data queries). #6577 ( proller )
  • Added optional message argument in throwIf . ( #5772 ) #6329 ( Vdimir )
  • Server exception got while sending insertion data is now being processed in client as well. #5891 #6711 ( dimarub2000 )
  • Added a metric DistributedFilesToInsert that shows the total number of files in filesystem that are selected to send to remote servers by Distributed tables. The number is summed across all shards. #6600 ( alexey-milovidov )
  • Move most of JOINs prepare logic from ExpressionAction/ExpressionAnalyzer to AnalyzedJoin . #6785 ( Artem Zuikov )
  • Fix TSan warning ‘lock-order-inversion’. #6740 ( Vasily Nemkov )
  • Better information messages about lack of Linux capabilities. Logging fatal errors with “fatal” level, that will make it easier to find in system.text_log . #6441 ( alexey-milovidov )
  • When enable dumping temporary data to the disk to restrict memory usage during GROUP BY , ORDER BY , it didn’t check the free disk space. The fix add a new setting min_free_disk_space , when the free disk space it smaller then the threshold, the query will stop and throw ErrorCodes::NOT_ENOUGH_SPACE . #6678 ( Weiqing Xu ) #6691 ( alexey-milovidov )
  • Removed recursive rwlock by thread. It makes no sense, because threads are reused between queries. SELECT query may acquire a lock in one thread, hold a lock from another thread and exit from first thread. In the same time, first thread can be reused by DROP query. This will lead to false “Attempt to acquire exclusive lock recursively” messages. #6771 ( alexey-milovidov )
  • Split ExpressionAnalyzer.appendJoin() . Prepare a place in ExpressionAnalyzer for MergeJoin . #6524 ( Artem Zuikov )
  • Added mysql_native_password authentication plugin to MySQL compatibility server. #6194 ( Yuriy Baranov )
  • Less number of clock_gettime calls; fixed ABI compatibility between debug/release in Allocator (insignificant issue). #6197 ( alexey-milovidov )
  • Move collectUsedColumns from ExpressionAnalyzer to SyntaxAnalyzer . SyntaxAnalyzer makes required_source_columns itself now. #6416 ( Artem Zuikov )
  • Add setting joined_subquery_requires_alias to require aliases for subselects and table functions in FROM that more than one table is present (i.e. queries with JOINs). #6733 ( Artem Zuikov )
  • Extract GetAggregatesVisitor class from ExpressionAnalyzer . #6458 ( Artem Zuikov )
  • system.query_log : change data type of type column to Enum . #6265 ( Nikita Mikhaylov )
  • Static linking of sha256_password authentication plugin. #6512 ( Yuriy Baranov )
  • Avoid extra dependency for the setting compile to work. In previous versions, the user may get error like cannot open crti.o , unable to find library -lc etc. #6309 ( alexey-milovidov )
  • More validation of the input that may come from malicious replica. #6303 ( alexey-milovidov )
  • Now clickhouse-obfuscator file is available in clickhouse-client package. In previous versions it was available as clickhouse obfuscator (with whitespace). #5816 #6609 ( dimarub2000 )
  • Fixed deadlock when we have at least two queries that read at least two tables in different order and another query that performs DDL operation on one of tables. Fixed another very rare deadlock. #6764 ( alexey-milovidov )
  • Added os_thread_ids column to system.processes and system.query_log for better debugging possibilities. #6763 ( alexey-milovidov )
  • A workaround for PHP mysqlnd extension bugs which occur when sha256_password is used as a default authentication plugin (described in #6031 ). #6113 ( Yuriy Baranov )
  • Remove unneeded place with changed nullability columns. #6693 ( Artem Zuikov )
  • Set default value of queue_max_wait_ms to zero, because current value (five seconds) makes no sense. There are rare circumstances when this settings has any use. Added settings replace_running_query_max_wait_ms , kafka_max_wait_ms and connection_pool_max_wait_ms for disambiguation. #6692 ( alexey-milovidov )
  • Extract SelectQueryExpressionAnalyzer from ExpressionAnalyzer . Keep the last one for non-select queries. #6499 ( Artem Zuikov )
  • Removed duplicating input and output formats. #6239 ( Nikolai Kochetov )
  • Allow user to override poll_interval and idle_connection_timeout settings on connection. #6230 ( alexey-milovidov )
  • MergeTree now has an additional option ttl_only_drop_parts (disabled by default) to avoid partial pruning of parts, so that they dropped completely when all the rows in a part are expired. #6191 ( Sergi Vladykin )
  • Type checks for set index functions. Throw exception if function got a wrong type. This fixes fuzz test with UBSan. #6511 ( Nikita Vasilev )

Performance Improvement ​

  • Optimize queries with ORDER BY expressions clause, where expressions have coinciding prefix with sorting key in MergeTree tables. This optimization is controlled by optimize_read_in_order setting. #6054 #6629 ( Anton Popov )
  • Allow to use multiple threads during parts loading and removal. #6372 #6074 #6438 ( alexey-milovidov )
  • Implemented batch variant of updating aggregate function states. It may lead to performance benefits. #6435 ( alexey-milovidov )
  • Using FastOps library for functions exp , log , sigmoid , tanh . FastOps is a fast vector math library from Michael Parakhin (Yandex CTO). Improved performance of exp and log functions more than 6 times. The functions exp and log from Float32 argument will return Float32 (in previous versions they always return Float64 ). Now exp(nan) may return inf . The result of exp and log functions may be not the nearest machine representable number to the true answer. #6254 ( alexey-milovidov ) Using Danila Kutenin variant to make fastops working #6317 ( alexey-milovidov )
  • Disable consecutive key optimization for UInt8/16 . #6298 #6701 ( akuzm )
  • Improved performance of simdjson library by getting rid of dynamic allocation in ParsedJson::Iterator . #6479 ( Vitaly Baranov )
  • Pre-fault pages when allocating memory with mmap() . #6667 ( akuzm )
  • Fix performance bug in Decimal comparison. #6380 ( Artem Zuikov )

Build/Testing/Packaging Improvement ​

Backward Incompatible Change ​

  • Removed rarely used table function catBoostPool and storage CatBoostPool . If you have used this table function, please write email to [email protected] . Note that CatBoost integration remains and will be supported. #6279 ( alexey-milovidov )
  • Disable ANY RIGHT JOIN and ANY FULL JOIN by default. Set any_join_distinct_right_table_keys setting to enable them. #5126 #6351 ( Artem Zuikov )

ClickHouse Release 19.13 ​

ClickHouse Release 19.13.6.51, 2019-10-02 ​

Bug Fix ​

  • This release also contains all bug fixes from 19.11.12.69.

ClickHouse Release 19.13.5.44, 2019-09-20 ​

Bug Fix ​

ClickHouse Release 19.13.4.32, 2019-09-10 ​

Bug Fix ​

Security Fix ​

  • Fix two vulnerabilities in codecs in decompression phase (malicious user can fabricate compressed data that will lead to buffer overflow in decompression). #6670 ( Artem Zuikov )

ClickHouse Release 19.13.3.26, 2019-08-22 ​

Bug Fix ​

  • Fix ALTER TABLE ... UPDATE query for tables with enable_mixed_granularity_parts=1 . #6543 ( alesapin )
  • Fix NPE when using IN clause with a subquery with a tuple. #6125 #6550 ( tavplubix )
  • Fixed an issue that if a stale replica becomes alive, it may still have data parts that were removed by DROP PARTITION. #6522 #6523 ( tavplubix )
  • Fixed issue with parsing CSV #6426 #6559 ( tavplubix )
  • Fixed data race in system.parts table and ALTER query. This fixes #6245 . #6513 ( alexey-milovidov )
  • Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address 0x14c0 that may happed due to concurrent DROP TABLE and SELECT from system.parts or system.parts_columns . Fixed race condition in preparation of mutation queries. Fixed deadlock caused by OPTIMIZE of Replicated tables and concurrent modification operations like ALTERs. #6514 ( alexey-milovidov )
  • Fixed possible data loss after ALTER DELETE query on table with skipping index. #6224 #6282 ( Nikita Vasilev )

Security Fix ​

  • If the attacker has write access to ZooKeeper and is able to run custom server available from the network where ClickHouse run, it can create custom-built malicious server that will act as ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Found by Eldar Zaitov, information security team at Yandex. #6247 ( alexey-milovidov )

ClickHouse Release 19.13.2.19, 2019-08-14 ​

New Feature ​

Backward Incompatible Change ​

  • The setting input_format_defaults_for_omitted_fields is enabled by default. Inserts in Distributed tables need this setting to be the same on cluster (you need to set it before rolling update). It enables calculation of complex default expressions for omitted fields in JSONEachRow and CSV* formats. It should be the expected behavior but may lead to negligible performance difference. #6043 ( Artem Zuikov ), #5625 ( akuzm )

Experimental Features ​

  • New query processing pipeline. Use experimental_use_processors=1 option to enable it. Use for your own trouble. #4914 ( Nikolai Kochetov )

Bug Fix ​

  • Kafka integration has been fixed in this version.
  • Fixed DoubleDelta encoding of Int64 for large DoubleDelta values, improved DoubleDelta encoding for random data for Int32 . #5998 ( Vasily Nemkov )
  • Fixed overestimation of max_rows_to_read if the setting merge_tree_uniform_read_distribution is set to 0. #6019 ( alexey-milovidov )

Improvement ​

  • Throws an exception if config.d file does not have the corresponding root element as the config file #6123 ( dimarub2000 )

Performance Improvement ​

Build/Testing/Packaging Improvement ​

ClickHouse Release 19.11 ​

ClickHouse Release 19.11.13.74, 2019-11-01 ​

Bug Fix ​

  • Fixed rare crash in ALTER MODIFY COLUMN and vertical merge when one of merged/altered parts is empty (0 rows). #6780 ( alesapin )
  • Manual update of SIMDJSON . This fixes possible flooding of stderr files with bogus json diagnostic messages. #7548 ( Alexander Kazakov )
  • Fixed bug with mrk file extension for mutations ( alesapin )

ClickHouse Release 19.11.12.69, 2019-10-02 ​

Bug Fix ​

ClickHouse Release 19.11.11.57, 2019-09-13 ​

  • Fix logical error causing segfaults when selecting from Kafka empty topic. #6902 #6909 ( Ivan )
  • Fix for function АrrayEnumerateUniqRanked with empty arrays in params. #6928 ( proller )

ClickHouse Release 19.11.10.54, 2019-09-10 ​

Bug Fix ​

  • Do store offsets for Kafka messages manually to be able to commit them all at once for all partitions. Fixes potential duplication in “one consumer - many partitions” scenario. #6872 ( Ivan )

ClickHouse Release 19.11.9.52, 2019-09-6 ​

Security Fix ​

  • If the attacker has write access to ZooKeeper and is able to run custom server available from the network where ClickHouse runs, it can create custom-built malicious server that will act as ClickHouse replica and register it in ZooKeeper. When another replica will fetch data part from malicious replica, it can force clickhouse-server to write to arbitrary path on filesystem. Found by Eldar Zaitov, information security team at Yandex. #6247 ( alexey-milovidov )

ClickHouse Release 19.11.8.46, 2019-08-22 ​

Bug Fix ​

  • Fix ALTER TABLE ... UPDATE query for tables with enable_mixed_granularity_parts=1 . #6543 ( alesapin )
  • Fix NPE when using IN clause with a subquery with a tuple. #6125 #6550 ( tavplubix )
  • Fixed an issue that if a stale replica becomes alive, it may still have data parts that were removed by DROP PARTITION. #6522 #6523 ( tavplubix )
  • Fixed issue with parsing CSV #6426 #6559 ( tavplubix )
  • Fixed data race in system.parts table and ALTER query. This fixes #6245 . #6513 ( alexey-milovidov )
  • Fixed wrong code in mutations that may lead to memory corruption. Fixed segfault with read of address 0x14c0 that may happed due to concurrent DROP TABLE and SELECT from system.parts or system.parts_columns . Fixed race condition in preparation of mutation queries. Fixed deadlock caused by OPTIMIZE of Replicated tables and concurrent modification operations like ALTERs. #6514 ( alexey-milovidov )

ClickHouse Release 19.11.7.40, 2019-08-14 ​

Bug Fix ​

  • Kafka integration has been fixed in this version.
  • Fix segfault when using arrayReduce for constant arguments. #6326 ( alexey-milovidov )
  • Fixed toFloat() monotonicity. #6374 ( dimarub2000 )
  • Fix segfault with enabled optimize_skip_unused_shards and missing sharding key. #6384 ( CurtizJ )
  • Fixed logic of arrayEnumerateUniqRanked function. #6423 ( alexey-milovidov )
  • Removed extra verbose logging from MySQL handler. #6389 ( alexey-milovidov )
  • Fix wrong behavior and possible segfaults in topK and topKWeighted aggregated functions. #6404 ( CurtizJ )
  • Do not expose virtual columns in system.columns table. This is required for backward compatibility. #6406 ( alexey-milovidov )
  • Fix bug with memory allocation for string fields in complex key cache dictionary. #6447 ( alesapin )
  • Fix bug with enabling adaptive granularity when creating new replica for Replicated*MergeTree table. #6452 ( alesapin )
  • Fix infinite loop when reading Kafka messages. #6354 ( abyss7 )
  • Fixed the possibility of a fabricated query to cause server crash due to stack overflow in SQL parser and possibility of stack overflow in Merge and Distributed tables #6433 ( alexey-milovidov )
  • Fixed Gorilla encoding error on small sequences. #6444 ( Enmk )

Improvement ​

ClickHouse Release 19.11.5.28, 2019-08-05 ​

Bug Fix ​

Backward Incompatible Change ​

  • Kafka still broken.

ClickHouse Release 19.11.4.24, 2019-08-01 ​

Bug Fix ​

  • Fix bug with writing secondary indices marks with adaptive granularity. #6126 ( alesapin )
  • Fix WITH ROLLUP and WITH CUBE modifiers of GROUP BY with two-level aggregation. #6225 ( Anton Popov )
  • Fixed hang in JSONExtractRaw function. Fixed #6195 #6198 ( alexey-milovidov )
  • Fix segfault in ExternalLoader::reloadOutdated(). #6082 ( Vitaly Baranov )
  • Fixed the case when server may close listening sockets but not shutdown and continue serving remaining queries. You may end up with two running clickhouse-server processes. Sometimes, the server may return an error bad_function_call for remaining queries. #6231 ( alexey-milovidov )
  • Fixed useless and incorrect condition on update field for initial loading of external dictionaries via ODBC, MySQL, ClickHouse and HTTP. This fixes #6069 #6083 ( alexey-milovidov )
  • Fixed irrelevant exception in cast of LowCardinality(Nullable) to not-Nullable column in case if it does not contain Nulls (e.g. in query like SELECT CAST(CAST('Hello' AS LowCardinality(Nullable(String))) AS String) . #6094 #6119 ( Nikolai Kochetov )
  • Fix non-deterministic result of “uniq” aggregate function in extreme rare cases. The bug was present in all ClickHouse versions. #6058 ( alexey-milovidov )
  • Segfault when we set a little bit too high CIDR on the function IPv6CIDRToRange . #6068 ( Guillaume Tassery )
  • Fixed small memory leak when server throw many exceptions from many different contexts. #6144 ( alexey-milovidov )
  • Fix the situation when consumer got paused before subscription and not resumed afterwards. #6075 ( Ivan ) Note that Kafka is broken in this version.
  • Clearing the Kafka data buffer from the previous read operation that was completed with an error #6026 ( Nikolay ) Note that Kafka is broken in this version.
  • Since StorageMergeTree::background_task_handle is initialized in startup() the MergeTreeBlockOutputStream::write() may try to use it before initialization. Just check if it is initialized. #6080 ( Ivan )

Build/Testing/Packaging Improvement ​

Backward Incompatible Change ​

  • Kafka is broken in this version.

ClickHouse Release 19.11.3.11, 2019-07-18 ​

New Feature ​

Bug Fix ​

  • Implement DNS cache with asynchronous update. Separate thread resolves all hosts and updates DNS cache with period (setting dns_cache_update_period ). It should help, when ip of hosts changes frequently. #5857 ( Anton Popov )
  • Fix segfault in Delta codec which affects columns with values less than 32 bits size. The bug led to random memory corruption. #5786 ( alesapin )
  • Fix segfault in TTL merge with non-physical columns in block. #5819 ( Anton Popov )
  • Fix rare bug in checking of part with LowCardinality column. Previously checkDataPart always fails for part with LowCardinality column. #5832 ( alesapin )
  • Avoid hanging connections when server thread pool is full. It is important for connections from remote table function or connections to a shard without replicas when there is long connection timeout. This fixes #5878 #5881 ( alexey-milovidov )
  • Support for constant arguments to evalMLModel function. This fixes #5817 #5820 ( alexey-milovidov )
  • Fixed the issue when ClickHouse determines default time zone as UCT instead of UTC . This fixes #5804 . #5828 ( alexey-milovidov )
  • Fixed buffer underflow in visitParamExtractRaw . This fixes #5901 #5902 ( alexey-milovidov )
  • Now distributed DROP/ALTER/TRUNCATE/OPTIMIZE ON CLUSTER queries will be executed directly on leader replica. #5757 ( alesapin )
  • Fix coalesce for ColumnConst with ColumnNullable + related changes. #5755 ( Artem Zuikov )
  • Fix the ReadBufferFromKafkaConsumer so that it keeps reading new messages after commit() even if it was stalled before #5852 ( Ivan )
  • Fix FULL and RIGHT JOIN results when joining on Nullable keys in right table. #5859 ( Artem Zuikov )
  • Possible fix of infinite sleeping of low-priority queries. #5842 ( alexey-milovidov )
  • Fix race condition, which cause that some queries may not appear in query_log after SYSTEM FLUSH LOGS query. #5456 #5685 ( Anton Popov )
  • Fixed heap-use-after-free ASan warning in ClusterCopier caused by watch which try to use already removed copier object. #5871 ( Nikolai Kochetov )
  • Fixed wrong StringRef pointer returned by some implementations of IColumn::deserializeAndInsertFromArena . This bug affected only unit-tests. #5973 ( Nikolai Kochetov )
  • Prevent source and intermediate array join columns of masking same name columns. #5941 ( Artem Zuikov )
  • Fix insert and select query to MySQL engine with MySQL style identifier quoting. #5704 ( Winter Zhang )
  • Now CHECK TABLE query can work with MergeTree engine family. It returns check status and message if any for each part (or file in case of simplier engines). Also, fix bug in fetch of a broken part. #5865 ( alesapin )
  • Fix SPLIT_SHARED_LIBRARIES runtime #5793 ( Danila Kutenin )
  • Fixed time zone initialization when /etc/localtime is a relative symlink like ../usr/share/zoneinfo/Asia/Istanbul #5922 ( alexey-milovidov )
  • clickhouse-copier: Fix use-after free on shutdown #5752 ( proller )
  • Updated simdjson . Fixed the issue that some invalid JSONs with zero bytes successfully parse. #5938 ( alexey-milovidov )
  • Fix shutdown of SystemLogs #5802 ( Anton Popov )
  • Fix hanging when condition in invalidate_query depends on a dictionary. #6011 ( Vitaly Baranov )

Improvement ​

  • Allow unresolvable addresses in cluster configuration. They will be considered unavailable and tried to resolve at every connection attempt. This is especially useful for Kubernetes. This fixes #5714 #5924 ( alexey-milovidov )
  • Close idle TCP connections (with one hour timeout by default). This is especially important for large clusters with multiple distributed tables on every server, because every server can possibly keep a connection pool to every other server, and after peak query concurrency, connections will stall. This fixes #5879 #5880 ( alexey-milovidov )
  • Better quality of topK function. Changed the SavingSpace set behavior to remove the last element if the new element have a bigger weight. #5833 #5850 ( Guillaume Tassery )
  • URL functions to work with domains now can work for incomplete URLs without scheme #5725 ( alesapin )
  • Checksums added to the system.parts_columns table. #5874 ( Nikita Mikhaylov )
  • Added Enum data type as a synonim for Enum8 or Enum16 . #5886 ( dimarub2000 )
  • Full bit transpose variant for T64 codec. Could lead to better compression with zstd . #5742 ( Artem Zuikov )
  • Condition on startsWith function now can uses primary key. This fixes #5310 and #5882 #5919 ( dimarub2000 )
  • Allow to use clickhouse-copier with cross-replication cluster topology by permitting empty database name. #5745 ( nvartolomei )
  • Use UTC as default timezone on a system without tzdata (e.g. bare Docker container). Before this patch, error message Could not determine local time zone was printed and server or client refused to start. #5827 ( alexey-milovidov )
  • Returned back support for floating point argument in function quantileTiming for backward compatibility. #5911 ( alexey-milovidov )
  • Show which table is missing column in error messages. #5768 ( Ivan )
  • Disallow run query with same query_id by various users #5430 ( proller )
  • More robust code for sending metrics to Graphite. It will work even during long multiple RENAME TABLE operation. #5875 ( alexey-milovidov )
  • More informative error messages will be displayed when ThreadPool cannot schedule a task for execution. This fixes #5305 #5801 ( alexey-milovidov )
  • Inverting ngramSearch to be more intuitive #5807 ( Danila Kutenin )
  • Add user parsing in HDFS engine builder #5946 ( akonyaev90 )
  • Update default value of max_ast_elements parameter #5933 ( Artem Konovalov )
  • Added a notion of obsolete settings. The obsolete setting allow_experimental_low_cardinality_type can be used with no effect. 0f15c01c6802f7ce1a1494c12c846be8c98944cd Alexey Milovidov

Performance Improvement ​

  • Increase number of streams to SELECT from Merge table for more uniform distribution of threads. Added setting max_streams_multiplier_for_merge_tables . This fixes #5797 #5915 ( alexey-milovidov )

Build/Testing/Packaging Improvement ​

Backward Incompatible Change ​

  • Kafka is broken in this version.
  • Enable adaptive_index_granularity = 10MB by default for new MergeTree tables. If you created new MergeTree tables on version 19.11+, downgrade to versions prior to 19.6 will be impossible. #5628 ( alesapin )
  • Removed obsolete undocumented embedded dictionaries that were used by Yandex.Metrica. The functions OSIn , SEIn , OSToRoot , SEToRoot , OSHierarchy , SEHierarchy are no longer available. If you are using these functions, write email to [email protected] . Note: at the last moment we decided to keep these functions for a while. #5780 ( alexey-milovidov )

ClickHouse Release 19.10 ​

ClickHouse Release 19.10.1.5, 2019-07-12 ​

New Feature ​

  • Add new column codec: T64 . Made for (U)IntX/EnumX/Data(Time)/DecimalX columns. It should be good for columns with constant or small range values. Codec itself allows enlarge or shrink data type without re-compression. #5557 ( Artem Zuikov )
  • Add database engine MySQL that allow to view all the tables in remote MySQL server #5599 ( Winter Zhang )
  • bitmapContains implementation. It’s 2x faster than bitmapHasAny if the second bitmap contains one element. #5535 ( Zhichang Yu )
  • Support for crc32 function (with behaviour exactly as in MySQL or PHP). Do not use it if you need a hash function. #5661 ( Remen Ivan )
  • Implemented SYSTEM START/STOP DISTRIBUTED SENDS queries to control asynchronous inserts into Distributed tables. #4935 ( Winter Zhang )

Bug Fix ​

  • Ignore query execution limits and max parts size for merge limits while executing mutations. #5659 ( Anton Popov )
  • Fix bug which may lead to deduplication of normal blocks (extremely rare) and insertion of duplicate blocks (more often). #5549 ( alesapin )
  • Fix of function arrayEnumerateUniqRanked for arguments with empty arrays #5559 ( proller )
  • Don’t subscribe to Kafka topics without intent to poll any messages. #5698 ( Ivan )
  • Make setting join_use_nulls get no effect for types that cannot be inside Nullable #5700 ( Olga Khvostikova )
  • Fixed Incorrect size of index granularity errors #5720 ( coraxster )
  • Fix Float to Decimal convert overflow #5607 ( coraxster )
  • Flush buffer when WriteBufferFromHDFS ’s destructor is called. This fixes writing into HDFS . #5684 ( Xindong Peng )

Improvement ​

Performance Improvement ​

  • Add the possibility to write the final mark at the end of MergeTree columns. It allows to avoid useless reads for keys that are out of table data range. It is enabled only if adaptive index granularity is in use. #5624 ( alesapin )
  • Improved performance of MergeTree tables on very slow filesystems by reducing number of stat syscalls. #5648 ( alexey-milovidov )
  • Fixed performance degradation in reading from MergeTree tables that was introduced in version 19.6. Fixes #5631. #5633 ( alexey-milovidov )

Build/Testing/Packaging Improvement ​

  • Implemented TestKeeper as an implementation of ZooKeeper interface used for testing #5643 ( alexey-milovidov ) ( levushkin aleksej )
  • From now on .sql tests can be run isolated by server, in parallel, with random database. It allows to run them faster, add new tests with custom server configurations, and be sure that different tests does not affect each other. #5554 ( Ivan )
  • Remove <name> and <metrics> from performance tests #5672 ( Olga Khvostikova )
  • Fixed “select_format” performance test for Pretty formats #5642 ( alexey-milovidov )

ClickHouse Release 19.9 ​

ClickHouse Release 19.9.3.31, 2019-07-05 ​

Bug Fix ​

  • Fix segfault in Delta codec which affects columns with values less than 32 bits size. The bug led to random memory corruption. #5786 ( alesapin )
  • Fix rare bug in checking of part with LowCardinality column. #5832 ( alesapin )
  • Fix segfault in TTL merge with non-physical columns in block. #5819 ( Anton Popov )
  • Fix potential infinite sleeping of low-priority queries. #5842 ( alexey-milovidov )
  • Fix how ClickHouse determines default time zone as UCT instead of UTC. #5828 ( alexey-milovidov )
  • Fix bug about executing distributed DROP/ALTER/TRUNCATE/OPTIMIZE ON CLUSTER queries on follower replica before leader replica. Now they will be executed directly on leader replica. #5757 ( alesapin )
  • Fix race condition, which cause that some queries may not appear in query_log instantly after SYSTEM FLUSH LOGS query. #5685 ( Anton Popov )
  • Added missing support for constant arguments to evalMLModel function. #5820 ( alexey-milovidov )

ClickHouse Release 19.9.2.4, 2019-06-24 ​

New Feature ​

  • Print information about frozen parts in system.parts table. #5471 ( proller )
  • Ask client password on clickhouse-client start on tty if not set in arguments #5092 ( proller )
  • Implement dictGet and dictGetOrDefault functions for Decimal types. #5394 ( Artem Zuikov )

Improvement ​

Bug Fix ​

  • Fix potential data loss in Kafka #5445 ( Ivan )
  • Fix potential infinite loop in PrettySpace format when called with zero columns #5560 ( Olga Khvostikova )
  • Fixed UInt32 overflow bug in linear models. Allow eval ML model for non-const model argument. #5516 ( Nikolai Kochetov )
  • ALTER TABLE ... DROP INDEX IF EXISTS ... should not raise an exception if provided index does not exist #5524 ( Gleb Novikov )
  • Fix segfault with bitmapHasAny in scalar subquery #5528 ( Zhichang Yu )
  • Fixed error when replication connection pool does not retry to resolve host, even when DNS cache was dropped. #5534 ( alesapin )
  • Fixed ALTER ... MODIFY TTL on ReplicatedMergeTree. #5539 ( Anton Popov )
  • Fix INSERT into Distributed table with MATERIALIZED column #5429 ( Azat Khuzhin )
  • Fix bad alloc when truncate Join storage #5437 ( TCeason )
  • In recent versions of package tzdata some of files are symlinks now. The current mechanism for detecting default timezone gets broken and gives wrong names for some timezones. Now at least we force the timezone name to the contents of TZ if provided. #5443 ( Ivan )
  • Fix some extremely rare cases with MultiVolnitsky searcher when the constant needles in sum are at least 16KB long. The algorithm missed or overwrote the previous results which can lead to the incorrect result of multiSearchAny . #5588 ( Danila Kutenin )
  • Fix the issue when settings for ExternalData requests couldn’t use ClickHouse settings. Also, for now, settings date_time_input_format and low_cardinality_allow_in_native_format cannot be used because of the ambiguity of names (in external data it can be interpreted as table format and in the query it can be a setting). #5455 ( Danila Kutenin )
  • Fix bug when parts were removed only from FS without dropping them from Zookeeper. #5520 ( alesapin )
  • Remove debug logging from MySQL protocol #5478 ( alexey-milovidov )
  • Skip ZNONODE during DDL query processing #5489 ( Azat Khuzhin )
  • Fix mix UNION ALL result column type. There were cases with inconsistent data and column types of resulting columns. #5503 ( Artem Zuikov )
  • Throw an exception on wrong integers in dictGetT functions instead of crash. #5446 ( Artem Zuikov )
  • Fix wrong element_count and load_factor for hashed dictionary in system.dictionaries table. #5440 ( Azat Khuzhin )

Build/Testing/Packaging Improvement ​

ClickHouse Release 19.8 ​

ClickHouse Release 19.8.3.8, 2019-06-11 ​

New Features ​

  • Added functions to work with JSON #4686 ( hcz ) #5124 . ( Vitaly Baranov )
  • Add a function basename, with a similar behaviour to a basename function, which exists in a lot of languages ( os.path.basename in python, basename in PHP, etc…). Work with both an UNIX-like path or a Windows path. #5136 ( Guillaume Tassery )
  • Added LIMIT n, m BY or LIMIT m OFFSET n BY syntax to set offset of n for LIMIT BY clause. #5138 ( Anton Popov )
  • Added new data type SimpleAggregateFunction , which allows to have columns with light aggregation in an AggregatingMergeTree . This can only be used with simple functions like any , anyLast , sum , min , max . #4629 ( Boris Granveaud )
  • Added support for non-constant arguments in function ngramDistance #5198 ( Danila Kutenin )
  • Added functions skewPop , skewSamp , kurtPop and kurtSamp to compute for sequence skewness, sample skewness, kurtosis and sample kurtosis respectively. #5200 ( hcz )
  • Support rename operation for MaterializeView storage. #5209 ( Guillaume Tassery )
  • Added server which allows connecting to ClickHouse using MySQL client. #4715 ( Yuriy Baranov )
  • Add toDecimal*OrZero and toDecimal*OrNull functions. #5291 ( Artem Zuikov )
  • Support Decimal types in functions: quantile , quantiles , median , quantileExactWeighted , quantilesExactWeighted , medianExactWeighted. #5304 ( Artem Zuikov )
  • Added toValidUTF8 function, which replaces all invalid UTF-8 characters by replacement character � (U+FFFD). #5322 ( Danila Kutenin )
  • Added format function. Formatting constant pattern (simplified Python format pattern) with the strings listed in the arguments. #5330 ( Danila Kutenin )
  • Added system.detached_parts table containing information about detached parts of MergeTree tables. #5353 ( akuzm )
  • Added ngramSearch function to calculate the non-symmetric difference between needle and haystack. #5418 #5422 ( Danila Kutenin )
  • Implementation of basic machine learning methods (stochastic linear regression and logistic regression) using aggregate functions interface. Has different strategies for updating model weights (simple gradient descent, momentum method, Nesterov method). Also supports mini-batches of custom size. #4943 ( Quid37 )
  • Implementation of geohashEncode and geohashDecode functions. #5003 ( Vasily Nemkov )
  • Added aggregate function timeSeriesGroupSum , which can aggregate different time series that sample timestamp not alignment. It will use linear interpolation between two sample timestamp and then sum time-series together. Added aggregate function timeSeriesGroupRateSum , which calculates the rate of time-series and then sum rates together. #4542 ( Yangkuan Liu )
  • Added functions IPv4CIDRtoIPv4Range and IPv6CIDRtoIPv6Range to calculate the lower and higher bounds for an IP in the subnet using a CIDR. #5095 ( Guillaume Tassery )
  • Add a X-ClickHouse-Summary header when we send a query using HTTP with enabled setting send_progress_in_http_headers . Return the usual information of X-ClickHouse-Progress, with additional information like how many rows and bytes were inserted in the query. #5116 ( Guillaume Tassery )

Improvements ​

  • Added max_parts_in_total setting for MergeTree family of tables (default: 100 000) that prevents unsafe specification of partition key #5166. #5171 ( alexey-milovidov )
  • clickhouse-obfuscator : derive seed for individual columns by combining initial seed with column name, not column position. This is intended to transform datasets with multiple related tables, so that tables will remain JOINable after transformation. #5178 ( alexey-milovidov )
  • Added functions JSONExtractRaw , JSONExtractKeyAndValues . Renamed functions jsonExtract<type> to JSONExtract<type> . When something goes wrong these functions return the correspondent values, not NULL . Modified function JSONExtract , now it gets the return type from its last parameter and does not inject nullables. Implemented fallback to RapidJSON in case AVX2 instructions are not available. Simdjson library updated to a new version. #5235 ( Vitaly Baranov )
  • Now if and multiIf functions do not rely on the condition’s Nullable , but rely on the branches for sql compatibility. #5238 ( Jian Wu )
  • In predicate now generates Null result from Null input like the Equal function. #5152 ( Jian Wu )
  • Check the time limit every (flush_interval / poll_timeout) number of rows from Kafka. This allows to break the reading from Kafka consumer more frequently and to check the time limits for the top-level streams #5249 ( Ivan )
  • Link rdkafka with bundled SASL. It should allow to use SASL SCRAM authentication #5253 ( Ivan )
  • Batched version of RowRefList for ALL JOINS. #5267 ( Artem Zuikov )
  • clickhouse-server: more informative listen error messages. #5268 ( proller )
  • Support dictionaries in clickhouse-copier for functions in <sharding_key> #5270 ( proller )
  • Add new setting kafka_commit_every_batch to regulate Kafka committing policy. It allows to set commit mode: after every batch of messages is handled, or after the whole block is written to the storage. It’s a trade-off between losing some messages or reading them twice in some extreme situations. #5308 ( Ivan )
  • Make windowFunnel support other Unsigned Integer Types. #5320 ( sundyli )
  • Allow to shadow virtual column _table in Merge engine. #5325 ( Ivan )
  • Make sequenceMatch aggregate functions support other unsigned Integer types #5339 ( sundyli )
  • Better error messages if checksum mismatch is most likely caused by hardware failures. #5355 ( alexey-milovidov )
  • Check that underlying tables support sampling for StorageMerge #5366 ( Ivan )
  • Сlose MySQL connections after their usage in external dictionaries. It is related to issue #893. #5395 ( Clément Rodriguez )
  • Improvements of MySQL Wire Protocol. Changed name of format to MySQLWire. Using RAII for calling RSA_free. Disabling SSL if context cannot be created. #5419 ( Yuriy Baranov )
  • clickhouse-client: allow to run with unaccessable history file (read-only, no disk space, file is directory, …). #5431 ( proller )
  • Respect query settings in asynchronous INSERTs into Distributed tables. #4936 ( TCeason )
  • Renamed functions leastSqr to simpleLinearRegression , LinearRegression to linearRegression , LogisticRegression to logisticRegression . #5391 ( Nikolai Kochetov )

Performance Improvements ​

  • Parallelize processing of parts of non-replicated MergeTree tables in ALTER MODIFY query. #4639 ( Ivan Kush )
  • Optimizations in regular expressions extraction. #5193 #5191 ( Danila Kutenin )
  • Do not add right join key column to join result if it’s used only in join on section. #5260 ( Artem Zuikov )
  • Freeze the Kafka buffer after first empty response. It avoids multiple invokations of ReadBuffer::next() for empty result in some row-parsing streams. #5283 ( Ivan )
  • concat function optimization for multiple arguments. #5357 ( Danila Kutenin )
  • Query optimisation. Allow push down IN statement while rewriting commа/cross join into inner one. #5396 ( Artem Zuikov )
  • Upgrade our LZ4 implementation with reference one to have faster decompression. #5070 ( Danila Kutenin )
  • Implemented MSD radix sort (based on kxsort), and partial sorting. #5129 ( Evgenii Pravda )

Bug Fixes ​

  • Fix push require columns with join #5192 ( Winter Zhang )
  • Fixed bug, when ClickHouse is run by systemd, the command sudo service clickhouse-server forcerestart was not working as expected. #5204 ( proller )
  • Fix http error codes in DataPartsExchange (interserver http server on 9009 port always returned code 200, even on errors). #5216 ( proller )
  • Fix SimpleAggregateFunction for String longer than MAX_SMALL_STRING_SIZE #5311 ( Azat Khuzhin )
  • Fix error for Decimal to Nullable(Decimal) conversion in IN. Support other Decimal to Decimal conversions (including different scales). #5350 ( Artem Zuikov )
  • Fixed FPU clobbering in simdjson library that lead to wrong calculation of uniqHLL and uniqCombined aggregate function and math functions such as log . #5354 ( alexey-milovidov )
  • Fixed handling mixed const/nonconst cases in JSON functions. #5435 ( Vitaly Baranov )
  • Fix retention function. Now all conditions that satisfy in a row of data are added to the data state. #5119 ( 小路 )
  • Fix result type for quantileExact with Decimals. #5304 ( Artem Zuikov )

Documentation ​

Build/Testing/Packaging Improvements ​

ClickHouse Release 19.7 ​

ClickHouse Release 19.7.5.29, 2019-07-05 ​

Bug Fix ​

ClickHouse Release 19.7.5.27, 2019-06-09 ​

New Features ​

  • Added bitmap related functions bitmapHasAny and bitmapHasAll analogous to hasAny and hasAll functions for arrays. #5279 ( Sergi Vladykin )

Bug Fixes ​

  • Fix segfault on minmax INDEX with Null value. #5246 ( Nikita Vasilev )
  • Mark all input columns in LIMIT BY as required output. It fixes ‘Not found column’ error in some distributed queries. #5407 ( Constantin S. Pan )
  • Fix “Column ‘0’ already exists” error in SELECT .. PREWHERE on column with DEFAULT #5397 ( proller )
  • Fix ALTER MODIFY TTL query on ReplicatedMergeTree . #5539 ( Anton Popov )
  • Don’t crash the server when Kafka consumers have failed to start. #5285 ( Ivan )
  • Fixed bitmap functions produce wrong result. #5359 ( Andy Yang )
  • Fix element_count for hashed dictionary (do not include duplicates) #5440 ( Azat Khuzhin )
  • Use contents of environment variable TZ as the name for timezone. It helps to correctly detect default timezone in some cases. #5443 ( Ivan )
  • Do not try to convert integers in dictGetT functions, because it does not work correctly. Throw an exception instead. #5446 ( Artem Zuikov )
  • Fix settings in ExternalData HTTP request. #5455 ( Danila Kutenin )
  • Fix bug when parts were removed only from FS without dropping them from Zookeeper. #5520 ( alesapin )
  • Fix segmentation fault in bitmapHasAny function. #5528 ( Zhichang Yu )
  • Fixed error when replication connection pool does not retry to resolve host, even when DNS cache was dropped. #5534 ( alesapin )
  • Fixed DROP INDEX IF EXISTS query. Now ALTER TABLE ... DROP INDEX IF EXISTS ... query does not raise an exception if provided index does not exist. #5524 ( Gleb Novikov )
  • Fix union all supertype column. There were cases with inconsistent data and column types of resulting columns. #5503 ( Artem Zuikov )
  • Skip ZNONODE during DDL query processing. Before if another node removes the znode in task queue, the one that did not process it, but already get list of children, will terminate the DDLWorker thread. #5489 ( Azat Khuzhin )
  • Fix INSERT into Distributed() table with MATERIALIZED column. #5429 ( Azat Khuzhin )

ClickHouse Release 19.7.3.9, 2019-05-30 ​

New Features ​

  • Allow to limit the range of a setting that can be specified by user. These constraints can be set up in user settings profile. #4931 ( Vitaly Baranov )
  • Add a second version of the function groupUniqArray with an optional max_size parameter that limits the size of the resulting array. This behavior is similar to groupArray(max_size)(x) function. #5026 ( Guillaume Tassery )
  • For TSVWithNames/CSVWithNames input file formats, column order can now be determined from file header. This is controlled by input_format_with_names_use_header parameter. #5081 ( Alexander )

Bug Fixes ​

  • Crash with uncompressed_cache + JOIN during merge (#5197) #5133 ( Danila Kutenin )
  • Segmentation fault on a clickhouse-client query to system tables. #5066 #5127 ( Ivan )
  • Data loss on heavy load via KafkaEngine (#4736) #5080 ( Ivan )
  • Fixed very rare data race condition that could happen when executing a query with UNION ALL involving at least two SELECTs from system.columns, system.tables, system.parts, system.parts_tables or tables of Merge family and performing ALTER of columns of the related tables concurrently. #5189 ( alexey-milovidov )

Performance Improvements ​

Documentation ​

Build/Testing/Packaging Improvements ​

 
推荐文章
豪气的感冒药  ·  MySQL :: MySQL Restrictions and Limitations :: 1 Restrictions on Stored Programs
1 周前
气宇轩昂的香瓜  ·  Copy and transform data to and from SQL Server - Azure Data Factory & Azure Synapse | Microsoft Lear
1 周前
干练的麻辣香锅  ·  使用 DataAdapter 更新数据源 - ADO.NET | Microsoft Learn
4 天前
逃跑的企鹅  ·  DataAdapter DataTable 和 DataColumn 對應 - ADO.NET | Microsoft Learn
4 天前
行走的苹果  ·  使用 Oracle 即时客户端运行数据泵导入或导出 | AWS re:Post
昨天
近视的针织衫  ·  关于规范网络转载版权秩序的通知-其他行政性规范文件-中国扫黄打非网
7 月前
逆袭的甘蔗  ·  Journal of Medical Internet Research - Performance Test of a Well-Trained Model for Meningioma Segme
9 月前
机灵的乌冬面  ·  西游记最厉害的兵器是什么？不是九齿耙，不是金箍棒，不是金刚琢_百科TA说
10 月前
大气的香槟  ·  如何正确地使用Matlab编码器进行动态和静态分配？
11 月前
考研的围巾  ·  docker一：容器介绍与lxc容器 - myworldworld - 博客园
1 年前
Link管理   ·   51好读   ·   Sov5搜索   ·   小百科
link管理 - 链接快照平台