mongodb - How to insert quickly to a very large collection -


i have collection of on 70 million documents. whenever add new documents in batches (lets 2k), insert operation slow. suspect because, mongo engine comparing _id's of new documents 70 million find out _id duplicate entries. since _id based index disk-resident, it'll make code lot slow.

is there anyway avoid this. want mongo take new documents , insert are, without doing check. possible?

diagnosing "slow" performance

your question includes number of leading assumptions how mongodb works. i'll address below, i'd advise try understand performance issues based on facts such database metrics (i.e. serverstatus, mongostat, mongotop), system resource monitoring, , information in mongodb log on slow queries. metrics need monitored on time can identify "normal" deployment, recommend using mongodb-specific monitoring tool such mms monitoring.

a few interesting presentations provide relevant background material performance troubleshooting , debugging are:

improving efficiency of inserts

aside understanding actual performance challenges lie , tuning deployment, improve efficiency of inserts by:

  • removing unused or redundant secondary indexes on collection

  • using bulk api insert documents in batches

assessing assumptions

whenever add new documents in batches (lets 2k), insert operation slow. suspect because, mongo engine comparing _id's of new documents 70 million find out _id duplicate entries. since _id based index disk-resident, it'll make code lot slow.

if collection has 70 million entries, not mean index lookup involves 70 million comparisons. indexed values stored in b-trees allow small number of efficient comparisons. exact number depend on depth of tree , how indexes built , value you're looking .. on order of 10s (not millions) of comparisons.

if you're curious internals, there experimental storage & index stats can enable in development environment: storage-viz: storage visualizers , commands mongodb.

since _id based index disk-resident, it'll make code lot slow.

mongodb loads working set (portion of data & index entries accessed) available memory.

if able create ids in approximately ascending order (for example, generated objectids) updates occur @ right side of b-tree , working set smaller (faq: "must working set fit in ram").

yes, can let mongo use _id itself, don't want waste index it. moreover, if let mongo generate _id won't need compare still duplicate key errors?

a unique _id required documents in mongodb. default objectid generated based on formula should ensure uniqueness (i.e. there extremely low chance of returning duplicate key exception, application not duplicate key exceptions , have retry new _id).

if have better candidate unique _id in documents, feel free use field (or collection of fields) instead of relying on generated _id. note _id immutable, shouldn't use fields might want modify later.


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based
Read more

thorough guide for profiling racket code -

did googling ("racket profiling", "racket measure performance"), didn't find , there no examples in docs. did google "profile" search on htdp - no luck. (profile (f ...)) output isn't obvious other small snippets. ideally want python's python -m cprofile usage examples. when search "python profiling", both duckduckgo , google give top result: 26.4. python profilers . (although scanned quickly, seems more of reference "thorough usage guide" examples. if had else in mind, perhaps please link that?) the equivalent racket documentation be: profile: statistical profiler . a usage example: #lang racket (require profile) (profile-thunk (thunk (function-to-profile arg0 arg1) )) here (thunk e) convenience (lambda () e) . bigger example: #lang racket (module mod racket (provide f) (define (f) (for/list ([i 10000]) i))) (require (prefix-in mod: 'mod)) (define (f) (for ([i 10000
Read more