Skip to content

[client-v2] Native format: any column containing a Tuple (Tuple/Map/Nested/Variant) is decoded row-wise and misread #3156

Description

@polyglotAI-bot

Description

NativeFormatReader decodes a non-Array column by calling the RowBinary value reader once per row, and an Array column by calling the RowBinary array-item reader once per row. RowBinary stores a Tuple inline, row by row. The Native format stores a Tuple as separate sub-columns, each holding all rows of one field (column-major).

So every column whose payload contains a tuple is read with the wrong layout. This includes top-level Tuple, Map (= Array(Tuple(k, v))), Nested, Variant, and a tuple nested inside an Array.

The result is silent data corruption when the byte count happens to match, and stream desynchronization (lost rows, spurious rows, or an exception) otherwise.

This is separate from #3088 (geo columns) and #3151 (block-boundary row loss).

Steps to reproduce

  1. Run any query below through the client-v2 binary reader with QuerySettings.setFormat(ClickHouseFormat.Native).
  2. Run the same query with RowBinaryWithNamesAndTypes and compare.
  3. Compare both to the server's own output (FORMAT JSONEachRow).

Observed, ClickHouse 26.9.1.1629, client 0.11.0-rc1-SNAPSHOT at 9029f68e9:

Query Server (JSONEachRow) RowBinaryWithNamesAndTypes Native
SELECT (toInt32(number+1), toInt32(number+101)) AS t FROM numbers(3) [1,101] [2,102] [3,103] [1,101] [2,102] [3,103] [1,2] [3,101] [102,103]
SELECT [(toInt32(1),toInt32(2)),(toInt32(3),toInt32(4))] AS a FROM numbers(1) [[1,2],[3,4]] [[1,2],[3,4]] [[1,3],[2,4]]
SELECT map('k', toInt32(number+1), 'j', toInt32(number+101)) AS m FROM numbers(3) {k:1,j:101} ... {k=1, j=101} ... {=1024}, {}, then 11 empty rows, then an exception
SELECT (concat('s', toString(number)), toInt32(number+7)) AS t FROM numbers(3) ['s0',7] ... [s0,7] [s1,8] [s2,9] ClientException: Failed to read block
SELECT if(number % 2 = 0, toInt32(number)::Variant(Int32,String), concat('v', toString(number))::Variant(Int32,String)) AS v FROM numbers(3) 0 'v1' 2 0 v1 2 0, 16777216, then IllegalArgumentException
SELECT toInt32(number+1) AS i FROM numbers(3) (control) 1 2 3 1 2 3 1 2 3

The first two rows are the dangerous ones: no error is raised, the row count is correct, and the values are wrong.

Error Log or Exception StackTrace

com.clickhouse.client.api.ClientException: Failed to read block          (Tuple(String, Int32))
java.lang.IllegalArgumentException: Non-empty typeName is required       (Variant(Int32, String))
com.clickhouse.client.api.ClientException: Reading  row 0                (Map(String, Int32))

Expected Behaviour

Native must return the same values as RowBinaryWithNamesAndTypes and as the server, for every type.

Server evidence that the layout is column-major — SELECT (toInt32(number+1), toInt32(number+101)) AS t FROM numbers(3) FORMAT Native:

01 03                                  1 column, 3 rows
01 74                                  name "t"
13 "Tuple(Int32, Int32)"               type
01 00 00 00  02 00 00 00  03 00 00 00  sub-column 1: 1, 2, 3
65 00 00 00  66 00 00 00  67 00 00 00  sub-column 2: 101, 102, 103

Each tuple field is written as a whole sub-column. The reader instead consumes 8 bytes per row from the start, producing (1,2) (3,101) (102,103).

Root cause

client-v2/src/main/java/com/clickhouse/client/api/data_formats/NativeFormatReader.java:

  • line 124-129 — the non-array branch calls binaryStreamReader.readValue(column) once per row. For Tuple this reaches BinaryStreamReader.readTuple (BinaryStreamReader.java:1073), which reads the fields of one tuple contiguously — the RowBinary layout.
  • line 109-123 — the array branch reads the offsets correctly (fixed in Fix client-v2 Native reader misreading multi-row Array columns #2956), then calls readArrayItem per row. That helper is also a RowBinary decoder, so Array(Tuple(...)) and Map(...) are flattened row-wise instead of sub-column-wise.

Variant has its own columnar discriminator layout in Native and is likewise not decoded.

Suggested fix

Extend the Native decoding to walk the type tree and read each sub-column over the whole block, in the same spirit as the geo decoder proposed in #3155 (BinaryStreamReader.readGeoNative):

  • Tuple(T1, ..., Tn) — read n sub-columns of nRows values each, then transpose into per-row tuples.
  • Array(T) — read the offsets (already correct), then read the flattened element sub-column as a single Native column of offsets[nRows-1] values, recursively.
  • Map(K, V) — the Array(Tuple(K, V)) case of the two rules above.
  • Nested — the tuple rule.
  • Variant / Dynamic — discriminators first, then one sub-column per variant.

Contrast cases that must keep their current behavior: scalar columns and Array of scalars already decode correctly (see the control row in the table) and the RowBinary* formats must be unaffected, since their row-wise layout is correct there.

If a full columnar decoder is too large for one change, rejecting the undecodable shapes with a clear exception — as readBlock already does for unsupported QBit shapes at line 95-108 — would at least stop the silent corruption.

Code Example

try (Client client = /* ... */;
     QueryResponse response = client.query(
             \"SELECT (toInt32(number+1), toInt32(number+101)) AS t FROM numbers(3)\",
             new QuerySettings().setFormat(ClickHouseFormat.Native)).get()) {
    ClickHouseBinaryFormatReader reader = client.newBinaryFormatReader(response);
    Map<String, Object> rec;
    while ((rec = reader.next()) != null) {
        System.out.println(Arrays.toString((Object[]) rec.get(\"t\")));
    }
}
// prints [1, 2] / [3, 101] / [102, 103]
// expected [1, 101] / [2, 102] / [3, 103]

Configuration

Environment

  • Cloud
  • Client version: 0.11.0-rc1-SNAPSHOT (main at 9029f68e9)
  • Language version: JDK 17
  • OS: Ubuntu 22.04 (container)

ClickHouse Server

  • ClickHouse Server version: 26.9.1.1629
  • ClickHouse Server non-default settings, if any: none
  • CREATE TABLE statements for tables involved: none, the queries are self-contained

Found by automated analysis of the client while working on #3088, and verified against a live server rather than by inspection.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions