python - Why influx performance is so slow -


i storing data in influx , quite confusing influx 4-5 times slow mysql. try test inserting 10000 rows in mysql , in influxdb. , stats below. 

mysql  real: 6m 39sec user: 2.956sec   sys: 0.504sec  influxdb  real: 6m 17.193sec user: 11.860sec sys: 0.328sec

my code influx given below, used same pattern store in mysql. 

#!/usr/bin/env python # coding: utf-8 import time import csv import sys import datetime import calendar import pytz influxdb import client influxdb datetime import datetime  host = 'localhost' port = 8086 user = "admin" password = "admin" db_name = "testdatabase" db = influxdb.influxdbclient(database=db_name)   def read_data():     open(file) f:         reader = f.readlines()[4:]        line in reader:             yield (line.strip().split(','))   fmt = '%y-%m-%d %h:%m:%s' file = '/home/rob/mycsvfile.csv'  csvtoinflux = read_data() body = [] metric in csvtoinflux:     timestamp = datetime.strptime(metric[0][1: len(metric[0]) - 1], fmt)      new_value = float(metric[1])     body.append({         'measurement': 'mytable1',         'time': timestamp,         'fields': {              'col1': metric[1],              'col2': metric[2],              'col3': metric[3],              'col4': metric[4],              'col5': metric[5],              'col6': metric[6],              'col7': metric[7],              'col8': metric[8],              'col9': metric[9]         }         })     db.write_points(body)

can give me idea how can improve it. think might due cache. cache option off default in influx db? , can guide me batch processing in influx. try on , google couldn't solve problem. newbie influx db. trying make faster. or tips.

inserting 1 one influxdb slow, should in batches. example, trying csv of 10000 lines (one one):

with open('/tmp/blah.csv') f:     lines = f.readlines()  import influxdb  inf = influxdb.influxdbclient('localhost', 8086, 'root', 'root', 'example1')  line in lines:     parts = line.split(',')     json_body = [{         'measurement': 'one_by_one',         'time': parts[0],         'fields':{             'my_value': int(parts[1].strip())         }     }]     inf.write_points(json_body)

this gives me result of

└─ $ ▶ time python influx_one.py  real    1m43.655s user    0m19.547s sys     0m3.266s

and doing small change insert lines of csv in 1 go:

json_body = [] line in lines:     parts = line.split(',')     json_body.append({         'measurement': 'one_batch',         'time': parts[0],         'fields':{             'my_value': int(parts[1].strip())         }     })  inf.write_points(json_body)

the result much better:

└─ $ ▶ time python influx_good.py  real    0m2.693s user    0m1.797s sys     0m0.734s

Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod...
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based ...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more