Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Teams

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

I am trying to use the mirrormaker tool to replicate data from one primary cluster to backup one, but I got following error.

nykpsr000001726$ bin/kafka-mirror-maker.sh --consumer.config config/mirror-consumer.properties --producer.config config/mirror-
producer.properties --whitelist my-replicated-topic
    [2017-02-03 06:17:00,193] FATAL [mirrormaker-thread-0] Mirror maker thread failure due to  (kafka.tools.MirrorMaker$MirrorMaker
    Thread)
    java.lang.IllegalArgumentException: Invalid timestamp -1
            at org.apache.kafka.clients.producer.ProducerRecord.<init>(ProducerRecord.java:60)
            at kafka.tools.MirrorMaker$defaultMirrorMakerMessageHandler$.handle(MirrorMaker.scala:678)
            at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:414)
    [2017-02-03 06:17:00,422] FATAL [mirrormaker-thread-0] Mirror maker thread exited abnormally, stopping the whole mirror maker.
bellow is the detail of configuration for consumer and producers.
mirror-consumer.properties
group.id=KafkaMirror-test-1
# consumer timeout should be -1 (default)
zookeeper.connect=ldnpsr000001131:2181
auto.offset.reset=smallest
mirror-producer.properties
bootstrap.servers=nykpsr000001726:9092
Really appriciate for a quick prompt.
                I am running Kafka 0.10 version, and using the third party test tool to generate testing messages. I believe the incompatibilty of pre 0.10 and 0.10 causing this issue. Again, thanks very much for your support @jose.goncabel
– Joey Trang
                Feb 3, 2017 at 16:43
It seems that you are facing a problem with the timestamps, the official documentation says:
  
Currently, Kafka Streams does not handle invalid (i.e., negative) timestamps returned from the TimestampExtractor gracefully, but fails with an exception, because negative timestamps cannot get handled in a meaningful way for any time based operators like window aggregates or joins.
  
Negative timestamp can occur for several reason.
  
You consume a topic that is written by old Kafka producer clients (i.e., version 0.9 or earlier), which don't use the new message format, and thus meta data timestamp field defaults to -1 if the topic is configured with log.message.timestamp.type=CreateTime
  
You consume a pre-0.10 topic after upgrading your Kafka cluster from 0.9 to 0.10: here all the data that was generated with 0.9 producers is not compatible with the 0.10 message format (and defaults to timestamp -1)
  
You consume a topic that is being written to by a third-party producer client that allows for embedding negative timestamps (KafkaProducer does check for negative timestamp an raises an exception for this case preventing invalid timestamp in the first place) 
  
The user provides a custom timestamp extractor that extracts a timestamp for the payload data (i.e., key-value pair), and this custom extractor might return negative timestamps.
Here you can read all the information.
        
Thanks for contributing an answer to Stack Overflow!
Please be sure to answer the question. Provide details and share your research!
But avoid …
Asking for help, clarification, or responding to other answers.
Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.