Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schema.generation.value.name - not taking effect in the schema registry #202

Open
esquare666 opened this issue Jul 5, 2022 · 0 comments

Comments

@esquare666
Copy link

esquare666 commented Jul 5, 2022

spool-schema-value.txt
I have given the value for this property "schema.generation.value.name": "nz.co.3es.example.customer.Value", and we have given the schema registry info in the worker level. However when I look at the schema, still the namespace is default value as com.github.jcustenborder.kafka.connect.model.Value
Connect worker config
-e CONNECT_VALUE_CONVERTER="io.confluent.connect.avro.AvroConverter"
-e CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL="https://3es.linux:8089" \

Connector config:
{
"connector.class": "com.github.jcustenborder.kafka.connect.spooldir.SpoolDirCsvSourceConnector",
"tasks.max": "1",
"topic": "ingest.spooldir.orders",
"input.path": "/tmp/data/unprocessed",
"finished.path": "/tmp/data/processed",
"error.path": "/tmp/data/error",
"input.file.pattern": ".*\.csv",
"schema.generation.enabled":"true",
"schema.generation.key.name": "nz.co.3es.example.customer.Key",
"schema.generation.value.name": "nz.co.3es.example.customer.Value",
"schema.generation.key.fields":"order_id",
"csv.first.row.as.header":"true",
"transforms":"castTypes",
"transforms.castTypes.type":"org.apache.kafka.connect.transforms.Cast$Value",
"transforms.castTypes.spec":"order_id:int32,customer_id:int32,order_total_usd:float32"
}'

From the logs.
key.schema = {"name":"com.github.jcustenborder.kafka.connect.model.Key","type":"STRUCT","isOptional":false,"fieldSchemas":{"order_id":{"type":"STRING","isOptional":true}}}
parser.timestamp.date.formats = [yyyy-MM-dd'T'HH:mm:ss, yyyy-MM-dd' 'HH:mm:ss]
parser.timestamp.timezone = UTC
processing.file.extension = .PROCESSING
schema.generation.enabled = true
schema.generation.key.fields = [order_id]
schema.generation.key.name = nz.co.3es.example.customer.Key
schema.generation.value.name = nz.co.3es.example.customer.Value
task.count = 1
task.index = 0
task.partitioner = ByName
timestamp.field =
timestamp.mode = PROCESS_TIME
topic = ingest.spooldir.orders
value.schema = {"name":"com.github.jcustenborder.kafka.connect.model.Value","type":"STRUCT","isOptional":false,"fieldSchemas":{"order_id":{"type":"STRING","isOptional":true},"customer_id":{"type":"STRING","isOptional":true},"order_total_usd":{"type":"STRING","isOptional":true},"make":{"type":"STRING","isOptional":true},"model":{"type":"STRING","isOptional":true},"delivery_city":{"type":"STRING","isOptional":true},"delivery_company":{"type":"STRING","isOptional":true},"delivery_address":{"type":"STRING","isOptional":true}}}

Is that mean, the name in the value.schema is used the default value even if we enable the "schema.generation.enabled":"true",

Refer the attached file for the schema from the registry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant