sql - Is Apache Camel's idempotent consumer pattern scalable? -


i'm using apache camel 2.13.1 poll database table have upwards of 300k rows in it. i'm looking use idempotent consumer eip filter rows have been processed.

i'm wondering though, whether implementation scalable or not. camel context is:-

<camelcontext xmlns="http://camel.apache.org/schema/spring">         <route id="main">         <from             uri="sql:select * transactions?datasource=mydatasource&amp;consumer.delay=10000&amp;consumer.useiterator=true" />         <transacted ref="propagation_required" />         <enrich uri="direct:invokeidempotenttransactions" />                         <!-- processors here executed on messages -->     </route>      <route id="idempotenttransactions">         <from uri="direct:invokeidempotenttransactions" />         <idempotentconsumer             messageidrepositoryref="jdbcidempotentrepository">             <ognl>#{request.body.id}</ognl>             <!-- here executed non-duplicates -->             <log message="non-duplicate" />             <to uri="stream:out" />         </idempotentconsumer>     </route>             </camelcontext>

it seem full 300k rows going processed every 10 seconds (via consumer.delay parameter) seems inefficient. expect sort of feedback loop part of pattern query feeds filter take advantage of set of rows processed.

however, messageid column in camel_messageprocessed table has pattern of

 {1908988=null}

where 1908988 request.body.id i've set eip key on doesn't make easy incorporate query.

is there better way of using camel_messageprocessed table feedback loop select statement sql server performing of load?

update:

so, i've since found out ognl code causing odd message id column value. changing to

<el>${in.body.id}</el>

has fixed it. so, have usable messageid column, can change 'from' sql query to

select * transactions tr tr.id in (select cmp.messageid camel_messageprocessed cmp cmp.processor = 'transactionprocessor')

but still think i'm corrupting idempotent consumer eip.

does else this? reason not to?

yes, is. need use scalable storage holding sets of processed messages. can use either hazelcast - http://camel.apache.org/hazelcast-idempotent-repository-tutorial.html or infinispan - http://java.dzone.com/articles/clustered-idempotent-consumer - depending on solution in stack. of course, jdbc repository work, if meets performance criteria selected.


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based
Read more

thorough guide for profiling racket code -

did googling ("racket profiling", "racket measure performance"), didn't find , there no examples in docs. did google "profile" search on htdp - no luck. (profile (f ...)) output isn't obvious other small snippets. ideally want python's python -m cprofile usage examples. when search "python profiling", both duckduckgo , google give top result: 26.4. python profilers . (although scanned quickly, seems more of reference "thorough usage guide" examples. if had else in mind, perhaps please link that?) the equivalent racket documentation be: profile: statistical profiler . a usage example: #lang racket (require profile) (profile-thunk (thunk (function-to-profile arg0 arg1) )) here (thunk e) convenience (lambda () e) . bigger example: #lang racket (module mod racket (provide f) (define (f) (for/list ([i 10000]) i))) (require (prefix-in mod: 'mod)) (define (f) (for ([i 10000
Read more