apache kafka - Spring Cloud Data Flow > Stream > Source throws UNKNOWN_TOPIC_OR_PARTITION when deploying in K8s but working in local deployment -


i'm trying run "hello, world" spring cloud data flow stream based on simple example explained @ http://cloud.spring.io/spring-cloud-dataflow/. i'm able create simple source , sink , run on local scdf server using kafka, until here correct.

now, i'm trying deploy in private cloud based on instructions listed @ http://docs.spring.io/spring-cloud-dataflow-server-kubernetes/docs/current-snapshot/reference/htmlsingle/#_getting_started. using deployment i'm able deploy simple "time | log" out-of-the-box stream no problems, example fails.

specific versions are:

  • docker version 1.13.1, build 092cba3
  • hyperkube 1.5.5
  • scdf 1.2.0.m2
  • zookeeper 3.4.9-1757313, built on 08/23/2016 06:50 gmt
  • kafka 0.10.1.1

source artifact logs are:

2017-04-06t11:05:07.429204866z 2017-04-06 11:05:07,428 info main-sendthread(10.0.0.181:2181) o.a.z.clientcnxn:876 - socket connection established 10.0.0.181/10.0.0.181:2181, initiating session 2017-04-06t11:05:07.440381666z 2017-04-06 11:05:07,439 info main-sendthread(10.0.0.181:2181) o.a.z.clientcnxn:1299 - session establishment complete on server 10.0.0.181/10.0.0.181:2181, sessionid = 0x15b155ef61e014a, negotiated timeout = 10000 2017-04-06t11:05:07.740130495z 2017-04-06 11:05:07,737 info main o.a.k.c.p.producerconfig:180 - producerconfig values: 2017-04-06t11:05:07.740160464z acks = 1 2017-04-06t11:05:07.740163408z batch.size = 16384 2017-04-06t11:05:07.740165226z block.on.buffer.full = false 2017-04-06t11:05:07.740166942z bootstrap.servers = [10.0.0.213:9092] 2017-04-06t11:05:07.740168741z buffer.memory = 33554432 2017-04-06t11:05:07.740170545z client.id = 2017-04-06t11:05:07.740172245z compression.type = none 2017-04-06t11:05:07.740173971z connections.max.idle.ms = 540000 2017-04-06t11:05:07.740175706z interceptor.classes = null 2017-04-06t11:05:07.744179899z reconnect.backoff.ms = 50 2017-04-06t11:05:07.744181600z request.timeout.ms = 30000 2017-04-06t11:05:07.744183356z retries = 0 2017-04-06t11:05:07.744185083z retry.backoff.ms = 100 2017-04-06t11:05:07.744186754z sasl.kerberos.kinit.cmd = /usr/bin/kinit 2017-04-06t11:05:07.744188494z sasl.kerberos.min.time.before.relogin = 60000 2017-04-06t11:05:07.744190205z sasl.kerberos.service.name = null 2017-04-06t11:05:07.744191916z sasl.kerberos.ticket.renew.jitter = 0.05 2017-04-06t11:05:07.744193763z sasl.kerberos.ticket.renew.window.factor = 0.8 2017-04-06t11:05:07.744195432z sasl.mechanism = gssapi 2017-04-06t11:05:07.744197163z security.protocol = plaintext 2017-04-06t11:05:07.744198789z send.buffer.bytes = 131072 2017-04-06t11:05:07.744200522z ssl.cipher.suites = null 2017-04-06t11:05:07.744202328z ssl.enabled.protocols = [tlsv1.2, tlsv1.1, tlsv1] 2017-04-06t11:05:07.744204161z ssl.endpoint.identification.algorithm = null 2017-04-06t11:05:07.744205837z ssl.key.password = null 2017-04-06t11:05:07.744207544z ssl.keymanager.algorithm = sunx509 2017-04-06t11:05:07.744212464z ssl.keystore.location = null 2017-04-06t11:05:07.744214272z ssl.keystore.password = null 2017-04-06t11:05:07.744216025z ssl.keystore.type = jks 2017-04-06t11:05:07.744217647z ssl.protocol = tls 2017-04-06t11:05:07.744219234z ssl.provider = null 2017-04-06t11:05:07.744220987z ssl.secure.random.implementation = null 2017-04-06t11:05:07.744222666z ssl.trustmanager.algorithm = pkix 2017-04-06t11:05:07.744224359z ssl.truststore.location = null 2017-04-06t11:05:07.744226022z ssl.truststore.password = null 2017-04-06t11:05:07.744228171z ssl.truststore.type = jks 2017-04-06t11:05:07.744230006z timeout.ms = 30000 2017-04-06t11:05:07.744231705z value.serializer = class org.apache.kafka.common.serialization.bytearrayserializer 2017-04-06t11:05:07.744233544z 2017-04-06t11:05:07.837193978z 2017-04-06 11:05:07,834 warn main o.a.k.c.p.producerconfig:188 - configuration 'key.deserializer' supplied isn't known config. 2017-04-06t11:05:07.837221870z 2017-04-06 11:05:07,835 warn main o.a.k.c.p.producerconfig:188 - configuration 'value.deserializer' supplied isn't known config. 2017-04-06t11:05:07.929207703z 2017-04-06 11:05:07,926 info main o.a.k.c.u.appinfoparser:83 - kafka version : 0.10.1.1 2017-04-06t11:05:07.929239636z 2017-04-06 11:05:07,927 info main o.a.k.c.u.appinfoparser:84 - kafka commitid : f10ef2720b03b247 2017-04-06t11:05:08.228817026z 2017-04-06 11:05:08,228 warn kafka-producer-network-thread | producer-1 o.a.k.c.networkclient:600 - error while fetching metadata correlation id 0 : {output=unknown_topic_or_partition} 2017-04-06t11:05:08.436574800z 2017-04-06 11:05:08,435 warn kafka-producer-network-thread | producer-1 o.a.k.c.networkclient:600 - error while fetching metadata correlation id 1 : {output=unknown_topic_or_partition}

and zookepeer logs are:

2017-04-06t11:04:38.000953447z 2017-04-06 11:04:38,000 [myid:] - info [processthread(sid:0 cport:2181)::preprequestprocessor@487] - processed session termination sessionid: 0x15b155ef61e0148 2017-04-06t11:05:04.939356606z 2017-04-06 11:05:04,938 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:nioservercnxnfactory@192] - accepted socket connection /10.1.98.5:48180 2017-04-06t11:05:04.940666418z 2017-04-06 11:05:04,939 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:zookeeperserver@928] - client attempting establish new session @ /10.1.98.5:48180 2017-04-06t11:05:04.943859474z 2017-04-06 11:05:04,943 [myid:] - info [syncthread:0:zookeeperserver@673] - established session 0x15b155ef61e0149 negotiated timeout 10000 client /10.1.98.5:48180 2017-04-06t11:05:07.325929074z 2017-04-06 11:05:07,325 [myid:] - info [processthread(sid:0 cport:2181)::preprequestprocessor@487] - processed session termination sessionid: 0x15b155ef61e0149 2017-04-06t11:05:07.342876962z 2017-04-06 11:05:07,341 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:nioservercnxn@1008] - closed socket connection client /10.1.98.5:48180 had sessionid 0x15b155ef61e0149 2017-04-06t11:05:07.429909440z 2017-04-06 11:05:07,429 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:nioservercnxnfactory@192] - accepted socket connection /10.1.98.5:48182 2017-04-06t11:05:07.429933377z 2017-04-06 11:05:07,429 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:zookeeperserver@928] - client attempting establish new session @ /10.1.98.5:48182 2017-04-06t11:05:07.441158222z 2017-04-06 11:05:07,439 [myid:] - info [syncthread:0:zookeeperserver@673] - established session 0x15b155ef61e014a negotiated timeout 10000 client /10.1.98.5:48182 2017-04-06t11:05:29.695276997z 2017-04-06 11:05:29,694 [myid:] - warn [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:nioservercnxn@357] - caught end of stream exception 2017-04-06t11:05:29.695325790z endofstreamexception: unable read additional data client sessionid 0x15b155ef61e014a, client has closed socket 2017-04-06t11:05:29.695328912z @ org.apache.zookeeper.server.nioservercnxn.doio(nioservercnxn.java:228) 2017-04-06t11:05:29.695331119z @ org.apache.zookeeper.server.nioservercnxnfactory.run(nioservercnxnfactory.java:203) 2017-04-06t11:05:29.695333009z @ java.lang.thread.run(thread.java:745) 2017-04-06t11:05:29.696333706z 2017-04-06 11:05:29,696 [myid:] - info [nioservercxn.factory:0.0.0.0/0.0.0.0:2181:nioservercnxn@1008] - closed socket connection client

i dont find log in kafka @ moment of exception.

code snippet source class is

package xxxx;  import java.text.simpledateformat; import java.util.date;  import org.springframework.boot.springapplication; import org.springframework.boot.autoconfigure.springbootapplication; import org.springframework.cloud.stream.annotation.enablebinding; import org.springframework.cloud.stream.messaging.source; import org.springframework.context.annotation.bean; import org.springframework.integration.annotation.inboundchanneladapter; import org.springframework.integration.core.messagesource; import org.springframework.messaging.support.genericmessage;  @springbootapplication @enablebinding(source.class) public class hellonitesapplication {     public static void main(string[] args)     {         springapplication.run(hellonitesapplication.class, args);     }      @bean     @inboundchanneladapter(value = source.output)     public messagesource<string> timermessagesource()     {         return () -> new genericmessage<>("hello " + new simpledateformat().format(new date()));     }

so, pod containing stream source keeps crashing in loop.

the problem seems fact property "spring.cloud.stream.bindings.output.destination=xxx" ignored implementation , deleted topic "output" before execution expected write in topic specified property.

after redeployed source works topic created properly, although inserting messages in "output" topic instead 1 specified property defined.


Comments

Post a Comment

Popular posts from this blog

inversion of control - Autofac named registration constructor injection -

does autofac support specifying registration name in components' constructors? example: ninject's namedattribute . you need use autofac.extras.attributed package on top achieve this so let's have 1 interface , 2 classes: public interface ihello { string sayhello(); } public class englishhello : ihello { public string sayhello() { return "hello"; } } public class frenchhello : ihello { public string sayhello() { return "bonjour"; } } then have consumer class, in want select instance injected: public class helloconsumer { private readonly ihello helloservice; public helloconsumer([withkey("en")] ihello helloservice) { if (helloservice == null) { throw new argumentnullexception("helloservice"); } this.helloservice = helloservice; } public string sayhello() { return this.helloservice.sayhello()...
Read more

verilog - Systemverilog dynamic casting issues -

i've code snippet following in testbench function void write_to_port( my_data_type_base data ); my_data_type_extended data_ext; if(!$cast(data_ext, data)); `uvm_error(get_type_name(), "failed cast"); `uvm_info(get_name(), $psprintf("data_ext :\n%s", data_ext.sprint()), uvm_medium) // write data_ext out port.... endfunction when run it, i'm getting uvm_error "failed cast." i'm not quire sure why $cast not returning 1. can see, i'm printing out extended class data item after casting uvm_info. can see it's being cast properly. if don't use $cast if condition, don't runtime error. isn't coding practice use if dynamic cast check if $cast returning 1 ? what might reason behind cast not returning 1 in above case? i think semicolon on line 'if' not belong? i think consumes if statement, , uvm_error executes regardless of how if evaluates: if(!$cast(data_ext, data)); <- no semico...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more