Avro "single-object encoding" decoding fail: `JsonParseException: Invalid length indicator for String: -98`

[birTiwana]

I have created a very simple avro schema file:

{
   "name":"SampleRecordDto",
   "type":"record",
   "namespace":"com.api.jsonata4java.avro.v1",
   "fields":[
      {
         "name":"name",
         "type":"string"
      }
   ]
}

Then generating a SampleRecordDto file from this using the following avro-maven-plugin.

			<plugin>
				<groupId>org.apache.avro</groupId>
				<artifactId>avro-maven-plugin</artifactId>
				<version>1.8.2</version>
				<executions>
					<execution>
						<phase>generate-sources</phase>
						<goals>
							<goal>schema</goal>
						</goals>
						<configuration>
							<sourceDirectory>${project.basedir}/src/main/avro/</sourceDirectory>
							<outputDirectory>${project.build.directory}/generated-sources/avro/</outputDirectory>
						</configuration>
					</execution>
				</executions>
			</plugin>

After then using jackson-dataformat-avro to convert an object of class SampleRecodDto to JSONNode using the following code:

package com.api.jsonata4java.test.expressions;

import com.api.jsonata4java.avro.v1.SampleRecordDto;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.dataformat.avro.AvroMapper;
import com.fasterxml.jackson.dataformat.avro.AvroSchema;
import org.apache.avro.Schema;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;

public class AvroTest {
    private static final AvroMapper avroMapper = new AvroMapper();
    public static void main(String[] args) throws IOException {
        SampleRecordDto sampleRecordDto = SampleRecordDto.newBuilder()
                .setName("apm")
                .build();

        JsonNode jsonNode = decode(sampleRecordDto.getSchema(), sampleRecordDto.toByteBuffer().array());
        System.out.println(jsonNode);

    }

    public  static JsonNode decode(Schema schema, byte[] message) {
        try (InputStream input = new ByteArrayInputStream(message)) {
            return avroMapper.reader(new AvroSchema(schema)).readTree(input);
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }
}

But I am running into the following exception:

Exception in thread "main" java.io.UncheckedIOException: com.fasterxml.jackson.core.JsonParseException: Invalid length indicator for String: -98
	at com.api.jsonata4java.test.expressions.AvroTest.decode(AvroTest.java:31)
	at com.api.jsonata4java.test.expressions.AvroTest.main(AvroTest.java:22)
Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid length indicator for String: -98
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2418)
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:749)
	at com.fasterxml.jackson.dataformat.avro.deser.JacksonAvroParserImpl.decodeString(JacksonAvroParserImpl.java:574)
	at com.fasterxml.jackson.dataformat.avro.deser.JacksonAvroParserImpl.decodeStringToken(JacksonAvroParserImpl.java:565)
	at com.fasterxml.jackson.dataformat.avro.deser.ScalarDecoder$StringReader$FR.readValue(ScalarDecoder.java:317)
	at com.fasterxml.jackson.dataformat.avro.deser.RecordReader$Std.nextToken(RecordReader.java:142)
	at com.fasterxml.jackson.dataformat.avro.deser.AvroParserImpl.nextToken(AvroParserImpl.java:97)
	at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeContainerNoRecursion(JsonNodeDeserializer.java:539)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:98)
	at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:23)
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
	at com.fasterxml.jackson.databind.ObjectReader._bindAsTree(ObjectReader.java:2149)
	at com.fasterxml.jackson.databind.ObjectReader._bindAndCloseAsTree(ObjectReader.java:2117)
	at com.fasterxml.jackson.databind.ObjectReader.readTree(ObjectReader.java:1794)
	at com.api.jsonata4java.test.expressions.AvroTest.decode(AvroTest.java:29)

Can I please get some help on this?

[cowtowncoder]

One thing to check, I think, is to ensure that this:

sampleRecordDto.toByteBuffer().array()

returns valid encoded document where content is at 0-offset. ByteBuffer could return backing array that has different offset; so checking what ByteBuffer.arrayOffset() returns would be good -- if it's not 0 there'd be a problem.
In general there is some discrepancy between data being decoded and schema; Avro as a format is very fragile unfortunately (as it thrives to be as compact as possible, little redundancy) and it is very easy to get corrupt data.

Also make sure to use a recent Jackson version; I don't think that is necessarily the problem here but sometimes bugs are reported against very old version and fix exists in newer versions.

[birTiwana]

I checked and the value ByteBuffer.arrayOffset() is actually 0.

[cowtowncoder]

Ok: apparently the root cause is as follows:

Root cause: Apache Avro 1.8+ uses Single Object Encoding in SpecificRecord.toByteBuffer(). The format
  adds a 10-byte header before the actual payload:
  - Bytes 0–1: magic 0xC3 0x01
  - Bytes 2–9: 8-byte Rabin schema fingerprint
  - Remaining: standard Avro binary payload

  When Jackson tries to parse this as raw Avro binary, it reads the first two bytes 0xC3 0x01 as a zigzag
  VarInt string length. 0xC3 has its high bit set (continuation), so it reads both bytes and gets i = 195,
   which zigzag-decodes to (195 >>> 1) ^ -(195 & 1) = 97 ^ -1 = -98 → the error.

I need to read more about this. For now, will just add failing test as reproduction.