MATLAB - How to load and handle of a big TXT file (32GB) -


first os all, sorry english...

i know better way load , handle big txt file (around 32gb, matrix 83.000.000x66). tried experiments textscan, import (out of memory), fgets, fget1,.... except import approach, methods works take time (much more 1 week).

i aim use database execute sampling process and, after that, neural network learning behabiour.

someone know how import type of data faster? thinking make database dump in other format (instead txt), exemplo sql server , try handle data accessing database queries.

other doubt, after load data, can save in .mat format , handle format in experiments? other better idea?

thanks in advance.

it's impossible hold such big matrix (5,478,000,000 values) in workspace/memory (unless you've got tons of ram). file format (.mat or .csv) doesn't matter! definitly have use database (or split file in sevaral smaller ones , calculate step step (takes long too).

personaly, have experiances sqlite3 , did similar 1.47mio x 23 matrix/csv file. http://git.osuv.de/markus/sqlite-demo (remember csv2sqlite.m designed run gnu octave [19k seconds @ night ...well, bad scripted :) ]. after imported sqlite3 database, can access data need within 8-12 seconds (take in comment header of leistung.m).

if csv file straight, can import sqlite3 example:

┌─[markus@x121e]─[/tmp] └──╼ cat file.csv  0.9736834199195674,0.7239387515366997,0.3382008456696883 0.6963824911102146,0.8328410999877027,0.5863203843393815 0.2291736458336333,0.1427739134201017,0.8062332551565472 ┌─[markus@x121e]─[/tmp] └──╼ sqlite3 csv.db sqlite version 3.8.4.3 2014-04-03 16:53:12 enter ".help" usage hints. sqlite> create table csvtest (col1 text not null, col2 text not null, col3 text not null); sqlite> .separator "," sqlite> .import file.csv csvtest sqlite> select * csvtest; 0.9736834199195674,0.7239387515366997,0.3382008456696883 0.6963824911102146,0.8328410999877027,0.5863203843393815 0.2291736458336333,0.1427739134201017,0.8062332551565472 sqlite> select col1 csvtest; 0.9736834199195674 0.6963824911102146 0.2291736458336333

all done https://github.com/markuman/go-sqlite (matlab , octave compatible! guess no 1 me has ever used it!) however, recommand version 2-beta in branch 2 (git checkout -b 2 origin/2) running in coop mode (you'll hit max string length sqlite3 in ego mode). there's html doku version 2 too. http://go-sqlite.osuv.de/doc/


Comments

Post a Comment

Popular posts from this blog

commonjs - How to write a typescript definition file for a node module that exports a function? -

consider, toml node module can use: // toml.d.ts declare module toml { export function parse(value:string):any; } declare module "toml" { export = toml; } then: /// <reference path="../../../../../defs/toml/toml.d.ts"/> import toml = require('toml'); toml.parse(...); however, node module export single function, such 'glob' ( https://github.com/isaacs/node-glob ). the node usage of module be: var glob = require("glob") glob("*.js", {}, callback(err, files) { ... }); you'd naively expect this: // glob.d.ts declare function globs(paths:string, options:any, callback:{(err:any, files:string[]):void; ...but because typescripts 'import' semantics bit strange, seems can use 'import .. = require()' statement alias modules . trying call: /// <reference path="../../../../../defs/glob/glob.d.ts"/> import blog = require('glob'); results in: error ts2072: mod...
Read more

openid - Okta: Failed to get authorization code through API call -

i'm integrating okta own idp server using okta's api. i'm implementing authorization code flow following steps below: in own server, use /api/v1/authn endpoint sessiontoken. use sessiontoken obtain authorization calling endpoint: /oauth2/v1/authorize?client_id=" + clientid + "&sessiontoken=" + sessiontoken + "&response_type=code&response_mode=query&scope=openid&redirect_uri=" + redirecturl + "&state=evanyang&nonce=" it's supposed return response status code 302 , location header containing redirect url code value. however, keep getting response status code 200 , without location header, html body saying "you using unsupported browser." , "javascript disabled on browser." according api documentation: http://developer.okta.com/docs/api/resources/oidc.html#authentication-request , sessiontoken parameter sufficient this: an okta one-time sessiontoken. allows api-based ...
Read more

ios - Change Storyboard View using Seague -

i have following code checks connection server, , depending on want change views if no connection found. nsstring *connect = [nsstring stringwithcontentsofurl:[nsurl urlwithstring:@"http://myserver.com"] encoding:nsutf8stringencoding error:nil]; if (connect == null) { [self performseguewithidentifier:@"network_failure" sender:self]; } else { // other code here } my problem not change views. have added storyboard segue between 2 views identifier of network_failure . not have navigation controller or that, problem? thanks help! no not problem. first of all, in objectivec exists nil object oriented null . must use that. so: if(connect == nil) or better: if(!connect) so due fact nil not same of null , probably code does't enter in if scope . you have been able discover breakpoint. anyway: be sure set trigger segue (line) 2 view controller , , name network_failure (that not best name. should instead view1toview2 ) as, m...
Read more