python - Parallel processing with Numpy.loadtxt() -


i have >100mb file needs read numpy.loadtxt()

the reading part main bottleneck in code. 72mb file takes 17.3s

is somehow possible read in parallel way file using loadtxt()

if possible without splitting file.

it looks numpy.loadtxt() problem.

http://wesmckinney.com/blog/?p=543

http://codrspace.com/durden/performance-lessons-for-reading-ascii-files-into-numpy-arrays/

according these sites, you're better off not using numpy's load function @ all.

pandas.read_csv , read_table should helpful pandas module.


Comments

Post a Comment

Popular posts from this blog

css - I want to align grid in center -

as see code grid_2 has float left. added center class fix in center div block floating left. want place grid_2 div in center. can 1 me fix this? html code: <div class="center"><div class="grid_2">content goes here </div> </div> css: .grid_0 { width: 100%; margin-bottom: 15px; display: block; } .grid_1, .grid_2, .grid_3, .grid_3c, .grid_4, .grid_4c { margin-left : 1.00%; margin-right : 1.00%; margin-bottom : 15px; float : left; display : block; -webkit-box-sizing: border-box; /* safari/chrome, other webkit */ -moz-box-sizing: border-box; /* firefox, other gecko */ box-sizing: border-box; /* opera/ie 8+ */ } .center{ display: block; text-align: center; margin: 0 auto !important; } .grid_1 { width: 98%; } .grid_2 { width: 48%; } please find link below http://jsfiddle.net/kiranm/aqbs6/5/ ok, here problem lies in width of grid_2. as width of grid_2 = 48% of...
Read more

python - SWIG function not printing output -

i'm having prblem in executing function i've written in c. i've made , interface file given below: %module cnode %{ #include "cnode.h" %} %include "cnode.h" and here c program : #include <stdio.h> #include "cnode.h" void kill(void) { printf("method executed\n"); } the header file: void kill(void); when i'm trying execute in python interpreter, don't output. there problem declarations or bug? figured out, kill() predefined function won't work. changing name of function works fine.
Read more

oop - Function for all prototypes - Javascript -

i'm working on video player video.js. on side i've got elements (volume, fullscreen, play, pause...) prototypes . on other side i've got functions parse() going use of canvas prototypes (they come different elements). parse function : vjs.parsehex = function(c) { if(c.charat(0) == "#") { return c.substring(1,7); } else if(c.substr(0,2) == "0x") { return c.substring(2,8); } else { return c; } }, vjs.parsered = function(c) { return parseint((this.parsehex(c)).substring(0,2),16); }, vjs.parsegreen = function(c) { return parseint((this.parsehex(c)).substring(2,4),16); }, vjs.parseblue = function(c) { return parseint((this.parsehex(c)).substring(4,6),16); } an exemple of canvas prototype : js.volumebar.prototype.drawsound = function(){ . . . } i want parse functions accessible of different prototypes. possible ? if yes, how can that? thank helping me, léa. it seems vjs sort...
Read more