sql - Is Apache Camel's idempotent consumer pattern scalable? -


i'm using apache camel 2.13.1 poll database table have upwards of 300k rows in it. i'm looking use idempotent consumer eip filter rows have been processed.

i'm wondering though, whether implementation scalable or not. camel context is:-

<camelcontext xmlns="http://camel.apache.org/schema/spring">         <route id="main">         <from             uri="sql:select * transactions?datasource=mydatasource&amp;consumer.delay=10000&amp;consumer.useiterator=true" />         <transacted ref="propagation_required" />         <enrich uri="direct:invokeidempotenttransactions" />                         <!-- processors here executed on messages -->     </route>      <route id="idempotenttransactions">         <from uri="direct:invokeidempotenttransactions" />         <idempotentconsumer             messageidrepositoryref="jdbcidempotentrepository">             <ognl>#{request.body.id}</ognl>             <!-- here executed non-duplicates -->             <log message="non-duplicate" />             <to uri="stream:out" />         </idempotentconsumer>     </route>             </camelcontext>

it seem full 300k rows going processed every 10 seconds (via consumer.delay parameter) seems inefficient. expect sort of feedback loop part of pattern query feeds filter take advantage of set of rows processed.

however, messageid column in camel_messageprocessed table has pattern of

 {1908988=null}

where 1908988 request.body.id i've set eip key on doesn't make easy incorporate query.

is there better way of using camel_messageprocessed table feedback loop select statement sql server performing of load?

update:

so, i've since found out ognl code causing odd message id column value. changing to

<el>${in.body.id}</el>

has fixed it. so, have usable messageid column, can change 'from' sql query to

select * transactions tr tr.id in (select cmp.messageid camel_messageprocessed cmp cmp.processor = 'transactionprocessor')

but still think i'm corrupting idempotent consumer eip.

does else this? reason not to?

yes, is. need use scalable storage holding sets of processed messages. can use either hazelcast - http://camel.apache.org/hazelcast-idempotent-repository-tutorial.html or infinispan - http://java.dzone.com/articles/clustered-idempotent-consumer - depending on solution in stack. of course, jdbc repository work, if meets performance criteria selected.


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod...
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based ...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more