ApiSource Kafka - Unreported Kafka Errors

XMLWordPrintable

      This report pertains to a known issue with COManage recording failed Kafka connections in ApiSource poll jobs as complete with 0 records.  We have been reviewing the php-rdkafka library, and we're able to register an error callback handler, trigger librdkafka to fire callbacks by issuing a consumer poll after each topic consumeBatch call, and bubble errors from the callback handler to the COmanage job.

      For your consideration, I've attached a git patch file for KafkaServer.php that registers an error handler, aggregates all received error messages in a class-level array, and raises an exception with its message set to a concatenation of values in that array.  So far we haven't collected an aggregate error message longer than the 256 cm_co_jobs.finish_summary field character limit; increasing the length of this field may or may not make sense.

       

            Assignee:
            Benn Oshrin (internet2.edu)
            Reporter:
            Trevor Buchanan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: