How to Troubleshoot "Found nested structure in input data. Ensure that your input events are a flat struct of primitive fields" error in Confluent Cloud UI

Description

This article provides a guide to troubleshoot the following error encountered on Confluent Cloud UI for Fully Managed PostgresSink, MicrosoftSqlServerSink, and MySqlSink Connectors:

Found nested structure in input data. Ensure that your input events are a flat struct of primitive fields.

Applies To

Microsoft SqlServer Sink Connector

MySql Sink Connector

Postgres Sink Connector

Confluent Cloud

Cause

The error occurs because the connector expects the input data to be a flat structure with only primitive fields. Nested structures or arrays within your input data are not supported.

Resolution

To resolve this error, you need to flatten the input data structure so that it only contains primitive fields. Here are three options to achieve this:

Apply the Flatten SMT to your connector configuration. This transformation flattens the structure of the record, so arrays and inner objects are converted to top-level fields.

{
"transforms": "flatten",
"transforms.flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value",
"transforms.flatten.delimiter": "."
}

2. Handle Nested Arrays:

You can preprocess your data using Kafka Streams or ksqlDB to transform the data into a flat structure before sending it to the connector.

Refer to the following docs to flatten the data:

How to flatten Array with ksqlDB

Flatten Nested Data with ksqlDB

3. Drop Unnecessary Fields:

If there are nested fields that are not required, you can use the ReplaceField SMT to drop these fields from the record.

{
    "transforms": "dropNested",
    "transforms.dropNested.type": "org.apache.kafka.connect.transforms.ReplaceField$Value",
    "transforms.dropNested.blacklist": "nestedField1,nestedField2"
}