java - executorservice to read data from database in chuncks and run process on them -


i'm trying write process read data database , upload onto cloud datastore.

how can decide partition strategy of data? want query table in chunks , process each chunk in 10 threads. each thread send data individual node on 10 node cluster on cloud..

where in below multi threading code dataquery extract , send 10 concurrent requests uploading data cloud be? important not loose cursor database query has processed far incase of failure, in database once batch of 100 records have been processed example, should update record statuses pending done in database column record. job run every 10 min pull data source database.

public class caller {     public static void main(string[] args) {          executorservice executor = executors.newfixedthreadpool(10);            (int = 0; < 10; i++) {                 runnable worker = new domaincdcprocessor(i);                 executor.execute(worker);               }             executor.shutdown();             while (!executor.isterminated()) {             }             system.out.println("finished threads");         }       } 

do select before starting workers determine set or range (first id - last id) of primary ids of records process. give each worker sub-set or range-part work with. make sure workers never go outside there given set or range.


Comments

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

openid - Okta: Failed to get authorization code through API call -

thorough guide for profiling racket code -