How to handle “The connector doesn't support tombstone values and therefore requires records with a non-null Struct value and non-null Struct schema” Error

Description

This article provides a guide to troubleshoot the following error encountered on Confluent Cloud UI for Fully Managed PostgresSink and Microsoft SqlServer Sink Connectors:

The connector doesn't support tombstone values and therefore requires records with a non-null Struct value and non-null Struct schema.

Applies To

PostgresSink Connector

MicrosoftSqlServerSink Connector

Confluent Cloud

Cause

This error occurs because the connectors mentioned do not support tombstone values. Tombstone records in Kafka are records where the value field is null. They are typically used to signify that a particular key should be deleted in a compacted topic. The Fully managed PostgresSink/MicrosoftSqlServerSink connectors require every record to have a non-null Struct value and a non-null Struct schema.

However, it can be difficult to differentiate between a completely null value, rather than a string of "null". We can use the following KB to confirm the same:

How to check for Tombstone messages in a topic

Resolution

To resolve this issue, ensure that the records being sent to the mentioned connectors do not contain tombstone values. Here are some steps to follow:

1. Filter Null Values:

   If you have a tombstone record, either the TombstoneHandler or Filter SMT can be used to filter out tombstone values (values that are null). An option for handling these messages would be to use the TombstoneHandler SMT, which allows you to ignore these null/tombstone messages. This SMT is not shipped with Confluent Platform but can be downloaded via Confluent Hub using the instructions in the link.

   For example:

   {
       "transforms": "tombstoneHandlerExample",
       "transforms.tombstoneHandlerExample.type": "io.confluent.connect.transforms.TombstoneHandler"
   }

   Or:

   {
       "transforms": "Filter",
       "transforms.Filter.type": "org.apache.kafka.connect.transforms.Filter",
       "transforms.Filter.predicate": "isNull",
       "predicates": "isNull",
       "predicates.isNull.type": "org.apache.kafka.connect.transforms.predicates.RecordIsTombstone"
   }

This filters the records and allows the connector to continue running as well, in addition to the TombstoneHandler. This SMT may not log messages when records are skipped/ignored; however, it works.

2. Check Source Connectors: Ensure that the source connectors producing the data are not generating tombstone records. If they are, modify the configurations to exclude tombstone records.

By following these steps, you can ensure that the data being processed by the Fully managed PostgresSink and MicrosoftSqlServerSink Connectors meet their requirements for non-null Struct values and schemas, thereby eliminating the occurrence of the error