java - BufferedReader taking too long -


this question has answer here:

this read file faster not write it. have 150mb file has json object inside it. use following code read it:

string filename ="/tmp/filetoread"; bufferedreader reader = new bufferedreader(new inputstreamreader(new fileinputstream(filename), charset.forname("utf-8"))); decompressedstring = reader.readline(); reader.close(); jsonobject obj = new jsonobject(decompressedstring); jsonarray profiledata = obj.getjsonobject("profiledata").getjsonarray("children"); ....

it single line file , since json can't split ( or atleast think so). reading file gives me outofmemory error or tle. file takes more 7 secs read , results in tle since execution of whole code cannot go beyond 7 seconds. oom on decompressedstring = reader.readline();.

is there way can reduce memory used or time takes read completely?

you have several problems @ hand:

  1. you're preemptively parsing much.

    the error happens when read line since said "i oom on decompressedstring = reader.readline();".

    you should never try read data line line. bufferedreader.readline() block until you've read character \r or \n or sequence \r\n. when processing data of length, you're never sure you'll 1 of characters. also, you're never sure you'll of characters outside of data itself. string may long or malformed. don't ever pretend know format. bufferedreader.readline() must used when parsing, not when acquiring data.

  2. you're not using appropriate library use-case

    reading json important, yes, you're reading @ once. when creating json, might want build stream (one of inputstream, reader or nio's channel/buffer).

    currently you're making json string. huge one. can safely assume you're going require @ 1 point twice memory need. 1 time in string , 1 time in finalized object.

    to reduce that, use appropriate library can pass 1 of stream mentioned above. mentioned in comments following: gson, json.simple , jackson.

  3. your file may big anyways.

    if data , want acquire subset of (here, want under {"profiledata":{"children": <data>}}). have way much. how many elements exist @ same level profiledata? how many elements exist @ same level children? know? way much. not under profiledata.children useless. percentage of total data that? 50%? 90%? 99%?

    to solve this, want 1 of 2 things: want less data or want able focus request.

    if want less data, ask data provider give less: need. why more that? makes no sense. tell him , "i want less".

    if want focused data, use library allows both parse , reduce amount of data. might want have library lets this: "parse json , return processingdata.children element". unfortunately know no library it. if others do, please add comment or answer. apparently, gson able if use jsonreader , selectively use skipvalue().


Comments

Post a Comment

Popular posts from this blog

inversion of control - Autofac named registration constructor injection -

does autofac support specifying registration name in components' constructors? example: ninject's namedattribute . you need use autofac.extras.attributed package on top achieve this so let's have 1 interface , 2 classes: public interface ihello { string sayhello(); } public class englishhello : ihello { public string sayhello() { return "hello"; } } public class frenchhello : ihello { public string sayhello() { return "bonjour"; } } then have consumer class, in want select instance injected: public class helloconsumer { private readonly ihello helloservice; public helloconsumer([withkey("en")] ihello helloservice) { if (helloservice == null) { throw new argumentnullexception("helloservice"); } this.helloservice = helloservice; } public string sayhello() { return this.helloservice.sayhello()...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more

verilog - Systemverilog dynamic casting issues -

i've code snippet following in testbench function void write_to_port( my_data_type_base data ); my_data_type_extended data_ext; if(!$cast(data_ext, data)); `uvm_error(get_type_name(), "failed cast"); `uvm_info(get_name(), $psprintf("data_ext :\n%s", data_ext.sprint()), uvm_medium) // write data_ext out port.... endfunction when run it, i'm getting uvm_error "failed cast." i'm not quire sure why $cast not returning 1. can see, i'm printing out extended class data item after casting uvm_info. can see it's being cast properly. if don't use $cast if condition, don't runtime error. isn't coding practice use if dynamic cast check if $cast returning 1 ? what might reason behind cast not returning 1 in above case? i think semicolon on line 'if' not belong? i think consumes if statement, , uvm_error executes regardless of how if evaluates: if(!$cast(data_ext, data)); <- no semico...
Read more