log4j:ERROR with kafka-avro-console-producer

[hth4]

While following the installing instructions I run into the following error with the kafka-avro-console-producer:

kafka-avro-console-producer --broker-list localhost:9092 --topic example --property parse.key=true --property key.schema='{"type":"record",name":"key_schema","fields":[{"name":"id","type":"int"}]}' --property "key.separator=$" --property value.schema='{"type":"record","name":"value_schema","fields":[{"name":"id","type":"int"},{"name":"firstName","type":"string"},{"name":"lastName","type":"string"}]}'

log4j:ERROR Could not read configuration file from URL [file://root/confluent-6.1.1/bin/../etc/schema-registry/log4j.properties].
java.net.UnknownHostException: root
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.Socket.connect(Socket.java:607)
at sun.net.ftp.impl.FtpClient.doConnect(FtpClient.java:962)
at sun.net.ftp.impl.FtpClient.tryConnect(FtpClient.java:924)
at sun.net.ftp.impl.FtpClient.connect(FtpClient.java:1019)
at sun.net.ftp.impl.FtpClient.connect(FtpClient.java:1005)
at sun.net.www.protocol.ftp.FtpURLConnection.connect(FtpURLConnection.java:311)
at sun.net.www.protocol.ftp.FtpURLConnection.getInputStream(FtpURLConnection.java:417)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.(Log4jLoggerFactory.java:66)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48)
at kafka.utils.Log4jControllerRegistration$.(Logging.scala:25)
at kafka.utils.CommandLineUtils$.(CommandLineUtils.scala:28)
at kafka.tools.ConsoleProducer$ProducerConfig.(ConsoleProducer.scala:228)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:41)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
log4j:ERROR Ignoring configuration file [file://root/confluent-6.1.1/bin/../etc/schema-registry/log4j.properties].
log4j:WARN No appenders could be found for logger (kafka.utils.Log4jControllerRegistration$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at io.confluent.kafka.formatter.SchemaMessageReader.parseSchema(SchemaMessageReader.java:212)
at io.confluent.kafka.formatter.SchemaMessageReader.getSchema(SchemaMessageReader.java:224)
at io.confluent.kafka.formatter.SchemaMessageReader.init(SchemaMessageReader.java:158)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:43)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

Is there more to install then copying the connector jar files?

[avelanarius]

What is your CONFLUENT_HOME environment variable? echo $CONFLUENT_HOME

In what directory (full path) did you install Confluent? Is it a directory only accessible to a root user (and does kafka-avro-console-producer have those permissions)?

[hth4]

I installed my testing installation as root

CONFLUENT_HOME=/root/confluent-6.1.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin://root/confluent-6.1.1/bin:/root/confluent-hub/bin://root/confluent-cli

-rwxr-xr-x. 1 root root 1545 Mar 17 01:20 /root/confluent-6.1.1/bin/kafka-avro-console-producer

[avelanarius]

I think the problem is in your PATH, specifically:

//root/confluent-6.1.1/bin:/root/confluent-hub/bin://root/confluent-cli

// should be /. Due to such specification of path, schema-registry-run-class script ran by the kafka-avro-console-producer incorrectly builds a path:
https://github.com/confluentinc/schema-registry/blob/master/bin/schema-registry-run-class#L68

It tries to connect to file://root/confluent-6.1.1/bin/../etc/schema-registry/log4j.properties - invalid URI, instead of file:/root/confluent-6.1.1/bin/../etc/schema-registry/log4j.properties - valid URI. If you fix your PATH, it should start correctly (assuming kafka-avro-console-producer is started as root).

[hth4]

you are right! After fixing the path this error gone away. Thanks!
Now I get:

[2021-05-10 17:37:12,527] ERROR Could not parse Avro schema (io.confluent.kafka.schemaregistry.avro.AvroSchemaProvider:57)
org.apache.avro.SchemaParseException: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('n' (code 110)): was expecting double-quote to start field name
at [Source: (String)"{"type":"record",name":"key_schema","fields":[{"name":"id","type":"int"}]}"; line: 1, column: 19]
at org.apache.avro.Schema$Parser.parse(Schema.java:1396)
at org.apache.avro.Schema$Parser.parse(Schema.java:1382)
at io.confluent.kafka.schemaregistry.avro.AvroSchema.(AvroSchema.java:71)
at io.confluent.kafka.schemaregistry.avro.AvroSchemaProvider.parseSchema(AvroSchemaProvider.java:54)
at io.confluent.kafka.schemaregistry.SchemaProvider.parseSchema(SchemaProvider.java:63)
at io.confluent.kafka.formatter.SchemaMessageReader.parseSchema(SchemaMessageReader.java:212)
at io.confluent.kafka.formatter.SchemaMessageReader.getSchema(SchemaMessageReader.java:224)
at io.confluent.kafka.formatter.SchemaMessageReader.init(SchemaMessageReader.java:158)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:43)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('n' (code 110)): was expecting double-quote to start field name
at [Source: (String)"{"type":"record",name":"key_schema","fields":[{"name":"id","type":"int"}]}"; line: 1, column: 19]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:637)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddName(ReaderBasedJsonParser.java:1784)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextFieldName(ReaderBasedJsonParser.java:932)
at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:250)
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4189)
at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2586)
at org.apache.avro.Schema$Parser.parse(Schema.java:1394)
... 9 more
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at io.confluent.kafka.formatter.SchemaMessageReader.parseSchema(SchemaMessageReader.java:212)
at io.confluent.kafka.formatter.SchemaMessageReader.getSchema(SchemaMessageReader.java:224)
at io.confluent.kafka.formatter.SchemaMessageReader.init(SchemaMessageReader.java:158)
at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:43)
at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)

Did I made another stupid mistake?

[hth4]

found the typo myself ...

'{"type":"record",name": ...
'{"type":"record","name":...

many thanks for the fast response!

[haaawk]

Can we close the issue now, @hth4?