mongodb - How to insert quickly to a very large collection -


i have collection of on 70 million documents. whenever add new documents in batches (lets 2k), insert operation slow. suspect because, mongo engine comparing _id's of new documents 70 million find out _id duplicate entries. since _id based index disk-resident, it'll make code lot slow.

is there anyway avoid this. want mongo take new documents , insert are, without doing check. possible?

diagnosing "slow" performance

your question includes number of leading assumptions how mongodb works. i'll address below, i'd advise try understand performance issues based on facts such database metrics (i.e. serverstatus, mongostat, mongotop), system resource monitoring, , information in mongodb log on slow queries. metrics need monitored on time can identify "normal" deployment, recommend using mongodb-specific monitoring tool such mms monitoring.

a few interesting presentations provide relevant background material performance troubleshooting , debugging are:

improving efficiency of inserts

aside understanding actual performance challenges lie , tuning deployment, improve efficiency of inserts by:

  • removing unused or redundant secondary indexes on collection

  • using bulk api insert documents in batches

assessing assumptions

whenever add new documents in batches (lets 2k), insert operation slow. suspect because, mongo engine comparing _id's of new documents 70 million find out _id duplicate entries. since _id based index disk-resident, it'll make code lot slow.

if collection has 70 million entries, not mean index lookup involves 70 million comparisons. indexed values stored in b-trees allow small number of efficient comparisons. exact number depend on depth of tree , how indexes built , value you're looking .. on order of 10s (not millions) of comparisons.

if you're curious internals, there experimental storage & index stats can enable in development environment: storage-viz: storage visualizers , commands mongodb.

since _id based index disk-resident, it'll make code lot slow.

mongodb loads working set (portion of data & index entries accessed) available memory.

if able create ids in approximately ascending order (for example, generated objectids) updates occur @ right side of b-tree , working set smaller (faq: "must working set fit in ram").

yes, can let mongo use _id itself, don't want waste index it. moreover, if let mongo generate _id won't need compare still duplicate key errors?

a unique _id required documents in mongodb. default objectid generated based on formula should ensure uniqueness (i.e. there extremely low chance of returning duplicate key exception, application not duplicate key exceptions , have retry new _id).

if have better candidate unique _id in documents, feel free use field (or collection of fields) instead of relying on generated _id. note _id immutable, shouldn't use fields might want modify later.


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod...
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based ...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more