Question:
I am facing a problem where my Debezium SQL Server Source Connectors is not streaming CDC captures in real time but streams all events once per day at the same time19:07:29
. Below is a sample recordsource.ts_ms
and ts_ms
. source.ts_ms
is 15 June 2022 16:47:53.757
but ts_ms
is 15 June 2022 19:07:29.349
. The documentation mentions that it means there is a lag but doesnt give any more info.https://debezium.io/documentation/reference/connectors/sqlserver#sqlserver-change-event-values
In the source object, ts_ms indicates the time when a change was committed in the database. By comparing the value for payload.source.ts_ms with the value for payload.ts_ms, you can determine the lag between the source database update and Debezium.
Below is the connector config
Answer:
Hassan , without logs this is difficult to solve. Please check the following and post more logs- Check if the CDC is enabled on the tables
- Query the CDC table in SQL Server , There is a default cdc schema where changes are captured.
The only SMT is a case change , so expect not to have any issues there
Check the Kafka connect logs. From the question probably is getting deployed to k8s. Login into the POD
Do you have any other hints on the logs ?
Are there issues in the SQL Server ? sp_who2 , sp_whoisactive perhaps for any clue ?
Any network issues between kubernetes and the RDS SQL Server ?
While the time correlation may be valid , these would be the ways to find any issues and narrow down what you face
If you have better answer, please add a comment about this, thank you!