Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-37371] Postgres CDC incremental source fails to handle upper-case table and column names #3929

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

morozov
Copy link
Contributor

@morozov morozov commented Feb 22, 2025

Problem summary

There are two problems in the code:

  1. The table name passed to to_regclass() isn't quoted, so the upper-case letters get lower-cased.
  2. The column names used in snapshot queries aren't quoted, so the upper-case letters get lower-cased.

There's also a minor issue with the quote() implementation that it doesn't doubles the double quotes. So if the table name or column name contains double quote, it will result in a SQL syntax error.

Connector code changes

  1. Instead of quoting the table name passed to to_regclass(), pass the schema name and table name individually. This way, they are interpreted as literals don't don't get automatically lower-cased.
  2. Use prepared statement parameters in the query against pg_class. Otherwise, the table and schema name would have to be additionally quoted.
  3. Quote column names used in snapshot queries.
  4. Replace double quote with two double quotes in quote() per SQL standard.

Test changes

I don't think that handling upper-case name should be covered by a special test case. Ideally, it should be tested exhaustively, so I decided to modify an existing test. Prior to that:

  1. Building SQL queries in tests has been refactored to enable using upper-case name. The new TableId class represents a qualified table name (schema name + table name). Its toSql() method generates a valid SQL name that can be used to build queries.
  2. PostgresSourceITCase has been cleaned up to eliminate some warnings (e.g. raw usage of parameterized types or non-closed Closable).
  3. Tests have been updated to use upper-case table and column names.

@morozov morozov marked this pull request as ready for review February 22, 2025 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant