Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2010-03-07:

[0:03] * Dantman wonders if there is something similar to iwevent that can be used on eth0
[0:06] <kriskowal> tobie: got the fixed link? http://j.mp/dwBGcZ
[0:09] <tobie> Yes. TY. Looking at it now.
[0:09] <tobie> Are you giving that talk on a construction site?
[0:34] <kriskowal> tobie: SF. cars downstairs :P
[4:39] <Dantman> Heh, I had a bit of fun with this one http://pastie.org/857871
[4:44] <Dantman> ^_^ I'm going to have some fun later setting up a daemon to turn on synergy when at work, and ssh when plugged into my lan at home.
[18:39] <Dantman> T_T Java's nio buffers have a nice mechanism for dealing with partial writes, but they're not usable in a JS API unless we use an API that matches it.
[18:42] <Dantman> http://java.sun.com/javase/6/docs/api/java/nio/ByteBuffer.html#compact()
[18:57] <Dantman> Java's nio buffers have a position, limit, and capacity. Writes are relative to a position in the buffer. A write using a ByteBuffer might be partial, but if it is, the position will be incremented to the point where it stopped writing.
[18:59] <Dantman> So while the ByteBuffer .hasRemaining() you can continue calling .write to write the remaining, using the same buffer. And additionally there is a .compact() method which moves the content between the position and limit to the start, sets the position to the end of that content, and sets the limit to the capacity of the buffer.
[19:01] <Dantman> In other words... It's a method that allows you to read data into a buffer, write some of it, move the data that hasn't been written to the start, and make another read call the same way that'll only read data into the remaining portion of the buffer.
[19:01] <Dantman> ...and the next write will be writing a combination of the unwritten data, and the newly read data.
[19:02] <Dantman> Though the doc's examples seem off
[19:07] <Wes-> this would make more sense if there was no actual move
[19:07] <Dantman> Oooooh, ok the examples were fine, I just didn't understand what .flip() did
[19:07] <Wes-> what you're talking here is zero-copy read/write, a good performance improvement
[19:08] <Wes-> not dissimilar to the use cases I envisioned for the unicode charset conversion routines I suggested for binary/f
[19:09] <Dantman> Thing is the buffers are abstracted a bit... I have no clue if compact means that the data is actually copied, or if implementations find some way to leave the data in place while rearanging how they leave the sequence in memory or something.
[19:10] <Dantman> ^_^ the java.nio.*Buffer setup could only be better if they had built in COW.
[19:10] <Dantman> They even have buffers that share regions of another buffer
[19:11] <Dantman> As well as direct and non-direct allocated buffers.
[19:13] <Dantman> direct buffers try to avoid any high level buffer copying or whatnot, and instead actually perform as much of the IO operations on it as possible with native system calls.
[19:14] <Dantman> They have a higher allocation cost, and sit outside of the normal garbage collected heap; But they support memory mapping, and can have significant performance gains if you are doing calls on them that do a large amount of system calls.
[19:14] <ashb> the java stuff is too abstracted for my taste
[19:15] <ashb> like reading a file you usually create 3 or 4 different classes in a chain don't you?
[19:15] <Dantman> Depends...
[19:16] <Dantman> You talking about the classic: BufferedReader reader = new BufferedReader(new InputStreamReader(new FileInputStream(file), charset));?
[19:18] <ashb> that seems familiar, yeah
[19:19] <ashb> that kind of abstraciton sucks :)
[19:20] <Dantman> I'll agree that it's annoying for common operations.
[19:20] <Dantman> s/annoying/overboard/
[19:20] <ashb> which is what should be optimized
[19:20] <ashb> make it easy to do the easy things, and possible to do the hard/almost-impossible
[19:21] <Dantman> But layering streams actually makes programing easier sometimes.
[19:22] <Dantman> I could have probably made some of my code in Kommonwealth much easier by writing a FixedLengthInputStream which abstracts a certain range of an InputStream earlier.
[19:23] <Dantman> Actually there's a bit of brainstorming I was doing a few days ago that is a bit relevant.
[19:24] <Dantman> Didn't finish it: http://gist.github.com/324584
[19:25] <ashb> // Dummy module names, not final. Ignore my JS1.8isms
[19:25] <ashb> i wis spidermonkey worked like that
[19:26] <ashb> still have to do {Stream: Stream} there
[19:26] <Dantman> Heh....
[19:26] <Dantman> Wait, huh?
[19:26] <ashb> or maybe it only destructures arrays
[19:26] <ashb> lemme check
[19:26] <Dantman> I don't see why var {Stream} = will work in Rhino in 1.8 mode, but not SM in 1.8 mode.
[19:27] <ashb> oh no it does work
[19:27] <Dantman> It was SM's 1.8 that created that
[19:27] <ashb> random nit:
[19:27] <ashb> oh its multipart
[19:27] <ashb> no thats fine
[19:27] <ashb> nm
[19:27] <Dantman> I suppose the caveat is that destructuring was introduced in 1.7, but the specific shortened form of using var {Foo} instead of var {Foo:Foo} was introduced in 1.8
[19:29] <ashb> the var {x,y} = require('module') is where things really start to get nice
[19:29] <Dantman> ^_^ And Rhino will honour that... If you select 170 in unstable, var {Foo} will be a syntax error but var {Foo:Foo} will work... You have to use the incomplete 180 in unstable to use the shortened syntax.
[19:29] <Dantman> Mhmm
[19:29] <ashb> fwiw some kind of destructuring is a harmony proposal
[19:30] <Dantman> The only disappointment is you will run into a minor hitch with cycilic requre.
[19:30] <ashb> http://wiki.ecmascript.org/doku.php?id=harmony:destructuring
[19:30] <ashb> if it matters, yeah
[19:30] <ashb> cyclic require is quite a rare occurance
[19:31] <Dantman> Mhmm
[19:31] <ashb> 'Array destructuring is implemented in the Opera browser starting with Opera 8'
[19:31] <ashb> really? didn't know that
[19:32] <Dantman> Though, the way require is isolated makes it a little less rare, since sometimes you just have to do things cycilicly to get two classes that refer to each other to work.
[19:32] <Dantman> Not common, but not as rare as if there was an alternative to shoving all your code in a single file.
[19:33] <Dantman> That's why my old banana idea had a scope that was shared between the files in the package iirc.
[19:36] <Dantman> Rest and spread implementation would be nice.
[19:36] <Dantman> It's inexistance prompted me to work on a fn.newApply awhile ago.
[19:37] <Dantman> You have to do some /fun/ things to make it work with builtin classes.
[19:40] <Dantman> Wes-, if you like the performance possibilities in nio, you'll love the MappedByteBuffer which is memory mapped file data, using the same api as a standard ByteBuffer with one or two mmap specific methods...
[19:43] <Dantman> In any case java.nio actually looks quite nice, I'm almost considering trying to use it for future things instead of continuing with the old API.
[19:45] <Dantman> If possible I'll probably try to implement any file streaming modules using nio.
[19:48] <Dantman> ^_^ nio even has a way to copy data directly from one channel to another without even using a Buffer avoiding as much java land overhead as possible.
[19:53] <Dantman> for ( var position = 0, size: fromFileChannel.size(); position < size; ) { position += fromFileChannel.transferTo(position, 1024, toFileChannel); }
[19:54] <Dantman> If I understand that correctly
[19:58] <Dantman> Heh, that leaves me with an idea for background threaded operations.
[19:59] <Dantman> File.open(fileA, "rb").pipeTo(File.open(fileB, "wb"));
[20:01] <Dantman> C/C++ impls can do it how they want... But I java I could implement that as a command that spawns a new thread in the background which pipes the data from one channel to another.
[20:01] <Dantman> ^_^ It could even return a promise.
[20:04] <Dantman> PromiseHelper.waitForAll([[fileA1, fileB1], [fileA2, fileB2], ...].map(function(pair) { return File.open(fileA, "rb").pipeTo(File.open(fileB, "wb")); }));
[20:06] <Dantman> system.stdin.pipeTo(system.stdout); <-- Equiv to `cat`
[20:12] <Dantman> system.stdin.pipeTo((new GzipPipe(1024)).pipeTo(system.stdout));
[20:15] <Dantman> Stream.pipe(system.stdout, new GzipPipe(1024), system.stdout); // A more efficient extra that'll only use one Pipe.
[20:15] <Dantman> Heh, first stdout should be stdin
[20:20] <ashb> Dantman: nice monologue :)
[20:56] <Dantman> Wes-, ashb: Does SpiderMonkey have the ability to throw similarly to Java's OutOfMemoryError, or does it just die?
[20:56] <ashb> no it throws
[20:57] <ashb> not sure if its catchable ot a js program tho
[20:57] <ashb> it dies more cleanrly thatn letting the system OOM killer take it
[21:00] <Dantman> Hmm... ok, bit in the more native background... What cases would cause a oom when doing file IO? I mean which part of a read everything algorithm would cause oom... It's trying to allocate the larger buffer, right?
[21:02] <ashb> depends how the buffer is coded, but yeah
[21:03] * Dantman was thinking of specing a .read() to throw a js side OOM that the program could attempt to recover from instead of just killing the program.
[21:04] <Dantman> In Java all I'd need to do would be to catch the OOM error and turn it into a nice JS error. Just wanted to know if that could be done outside of Rhino.
[21:05] <ashb> not sure about v8
[21:09] <Dantman> ^_^ If v8 can't it could be dropped from spec... ;) But that still means I can support it completely in both versions of MonkeyScript.
[21:17] <Dantman> Oh, I did a bit of performance testing related to instancing File objects: http://pastie.org/858595
[21:19] <ashb> just out of curiosity do you get the same numbers in pure java?
[21:20] <ashb> also i can't work out what numbers are interesting there
[21:24] <Dantman> No clue
[21:24] <Dantman> Heh, Bharani: hmm. looks like js will become the new common lisp"
[21:25] <Dantman> lisp / common lisp | js / common js
[21:28] <Dantman> 100 thousand times created a file object and ran canRead in an average of 1746289075.5ns; created a single File object and ran 100 thousand canRead calls on it in an average of 1296298127.25ns
[21:29] <Dantman> Works out to about 0.0044999094825ms to create each File instance.
[21:30] <Dantman> Hmm... though, I suppose that's about 500ms to create 100000
[21:32] <Dantman> I guess the results are saying compared to the cost of running js calls and a system call the actual amount of time spend recreating wasted File instances are trivial enough they shouldn't really matter to an api.
[21:32] <Dantman> So I guess I'm dropping my objection to a path based level 0 api.
[21:33] <Dantman> However I'm unsure about my issue with permissions...
[21:36] <Dantman> Hmm... waitasecond, rather than a new thread for each .pipe or pipeTo call I should actually be able to do that with only a single extra daemon thread.
[21:38] <Dantman> Then again, that leaves me with the dilemma of how to avoid implied exit from happening.
[21:50] <Dantman> Perhaps I could use a lock; pipe thread locks when it goes into piping, and unlocks when finished. When leaving for exit main thread tries to lock, if it can't it waits till it can (ie: till pipe thread finishes pipe operation), if another thread already a lock when the pipe thread tries to do an operation, it kills itself.
[21:53] <ashb> locks are genrally slow and heavy
[21:53] <ashb> if oyu can avoid them its good practice to
[21:55] * Dantman just needs a way to get the main thread to wait on the daemon thread doing pipe operations if the main thread is going to go into the implied exit phase (ie: All when scripts have been executed, no async functions are waiting, and there are no non-daemon threads).
[21:56] <Dantman> Otherwise there will be a not so fun situation where if you startup a few pipe based copy operations and don't do a manual .wait on all of them, the system will exit before the pipe operations are finished.
[21:58] <Dantman> Heh, you know, that almost makes me want to make file.copy return a promise.
[21:58] <Dantman> PromiseHelper.waitForAll(file.copy(...), file.copy(...), file.copy(...), file.copy(...), file.copy(...), ...);
[22:00] <MisterN> Dantman: what about file.copy.promise(...) or something?
[22:01] <Dantman> Hmmm, maybe... I dunno, I was considering something more like an .async flag that would make it explicit.
[22:04] <Dantman> Though, then again there are only 3 not so nice ways to do that; Local to file instances (con: have to specify it for each instance); As File.async (con: too global, applies to the entire system, makes for confusing I didn't set this but someone else did cases); Or we create some strange Filesystem object you create new File classes from... Nowait... FileFactory!
[22:17] <Dantman> Agh, crap, I don't even know when the time to do the lock would be anyways...
[22:19] <Dantman> Heh, fun part about java.nio.Buffer is .slice() doesn't do quite what you think...
[22:22] <Dantman> .range() would roughly be implemented using buf.position(start_op).limit(stop_op).slice()
[22:22] <Dantman> Binary/F's range that is.
[22:23] * Dantman debates how much of the java.nio API we'd actually want in JS.
[22:25] <ashb> from the way you've been complaining: not much
[22:25] <Dantman> Complaining?
[22:26] <Dantman> The nio API is nice...
[22:27] <Dantman> The only thing is, I don't know how much we want to use nio's pattern for buffer to buffer copying.
[22:27] <Dantman> ((I mean, when you're not using transfer(To|From) which is optimized, for some reason))
[22:29] <Dantman> buf.clear(); while( in.read(buf) || buf.position > 0 ) { buf.flip(); out.write(buf); buf.compact(); }
[22:30] <Dantman> .clear() sets the position to 0 and the limit to the buffers capacity readying it for reading.
[22:31] <Dantman> The .read() ends with the position set to the end of what was read.
[22:31] <Dantman> .flip() sets position to 0 and limit to where position just was, reading it for .write
[22:31] <Dantman> That means that .write only writes in the data within the area of the buffer that was read into.
[22:32] <Dantman> Then .compact() moves any unwritten data to the beginning of the buffer, sets position to the end of that data, and limit to the capacity.
[22:32] <ashb> eugh
[22:32] <ashb> complex as all hell
[22:33] <Dantman> ;) Though you'll have to admit it's probably shorter to write than the actual operations it's performing in the background.
[22:33] <ashb> so? its more complex then it needs to be
[22:33] <Dantman> ;) That's what transferTo and transferFrom is for.
[22:34] <ashb> whats up with the flip and compact shit?
[22:34] <Dantman> Bad naming sense I suppose, they are confusing...
[22:35] <Dantman> nio Buffers have a notion of position, limit, capacity, and relative operations.
[22:36] <Dantman> capacity is the allocated size of the buffer in memory. position is the current position for relative operations. and limit is the point where relative operations stop.
[22:36] <Dantman> In other words, for reading, read only happens in between the position and limit.
[22:37] <Dantman> Hmm, maybe I misphrased that line.
[22:38] <Dantman> Hmm... Maybe it'll be easier if I translate that into what you would do with a byte[] instead.
[22:39] <Dantman> Note that this is a little bit fudged because the old java api that uses byte[] doesn't have a partial .write() method.
[22:52] <Dantman> ashb, http://pastie.org/858751 ;) Top part is nio, bottom part is pretty much the same operations without nio. (Pretending that the old has a partial .write); The later probably has some vm land overhead rather than more native operations, in addition to the code bloat.
[22:53] <Dantman> Course, you don't get insane things like that with the old api... Because the old api has no partial write, so you do things in a more inefficient way that result in shorter code.
[22:54] <Dantman> It may be ugly in some regards, but it's not something to be disregarded... Java has a very strong similarity to JS in this regard.
[22:57] <Dantman> Both Java and ES are sitting in a higher level land away from the low level native calls. So both Java and JS have to find a way around that to do as much of the repeat IO operations outside of the high level land to make them efficient.
[22:58] <Dantman> So java.nio is basically prior art attempting to do efficient IO in a high level land as we are.
[23:10] <Dantman> ashb, btw, about that unfinished brainstorm I posted earlier on. did you notice what I did with sockets and streams?
[23:10] <ashb> didn't look in any detail
[23:10] <Dantman> socket.open("t");
[23:10] <ashb> wtf?
[23:12] <Dantman> A idea of having a consistent API where for the various things like Files, Sockets, Pipes, etc... you get a stream using .open
[23:13] <ashb> seems like a false econmy to me.
[23:13] <ashb> at least wtf is "t" param?
[23:13] <Dantman> Oh, t=text, b=binary... Just like with file.open
[23:14] <ashb> like whose file.open exactly :)
[23:14] <Dantman> Heh
[23:14] <Dantman> pythons, rubys, and php's iirc
[23:15] <Dantman> Heh, java's RandomAccessFile actually comes close too iirc
[23:16] <Dantman> new RandomAccessFile(f, "rw");
[23:16] <ashb> lets not take an API lessons from Java, eh?
[23:16] <MisterN> Dantman: i don't think text sockets are well-defined.
[23:16] <ashb> also "text" isn't well defined
[23:16] <ashb> you need to provide a charset
[23:16] <Dantman> Heh, right...
[23:16] <MisterN> well fopen a file as text and chances are, the charset won't be handled
[23:16] <MisterN> but \r\n->\n might be done
[23:17] <ashb> fopen might not, but text-mode streams in commonJS probably should deal with charsets
[23:17] <MisterN> possibly
[23:17] <Dantman> ^_^ Ignore that part, oversight on my part I suppose.
[23:18] <MisterN> i don't think "t" should be used then. rather, use {text_encoding: "shift-jis"} or something
[23:18] <Dantman> I use binary methods on that stream anyways, t is out of place...
[23:18] <Dantman> Maybe I meant "r" instead?
[23:18] <MisterN> the whole option string thing is just for convenience IMHO
[23:18] <Dantman> mhmm
[23:19] <Dantman> Most people will probably opt to use it though.
[23:19] <MisterN> open("r") is more convenient than open({write:false, read:true})
[23:19] <Dantman> mhmm
[23:19] <Dantman> precisely why
[23:19] <MisterN> it's familiar because most people know the fopen syntax
[23:19] <Dantman> That, and it's familiar.
[23:19] <Dantman> Heh
[23:20] <MisterN> but "t" is not familar.
[23:20] <Dantman> mhmm
[23:20] <Dantman> Ignore t
[23:20] <MisterN> :)
[23:20] <Dantman> php probably came up with t
[23:20] <Dantman> ;) And we know how nice PHP is...
[23:21] <MisterN> it's even better than java
[23:22] <Dantman> Bah,
[23:24] <Dantman> Besides a few things related to it being compiled and classical, js has more in common with java than php.
[23:24] <Dantman> And those are the good parts...
[23:26] <ashb> *bad* parts
[23:26] <ashb> Date#getMonth() being 0 based. W.T.F.
[23:26] <ashb> bigest WTF ever invented
[23:27] <Dantman> Ok, a few bad parts too...
[23:27] <ashb> new Date(2010,2,7)
[23:29] <Dantman> ...Classes favoured over piles of wildly named functions; Strings that are actually text rather than binary with a not so defined charset which pretends to be text. The ability to write somewhat flexible inline code without using eval.
[23:40] <MisterN> javascript does NOT have strings that are actually text.
[23:41] <MisterN> it has strings that are binaries where the charset is partially defined.
[23:41] <Dantman> Meh, that's what I mean... At least the charset is actually defined, and the methods actually obey that.
[23:42] <Dantman> length of a string is /fun/ without the mb extension to php.
[23:42] <ashb> Dantman: its *not* defined
[23:42] <ashb> JS strings are explicitly NOT UTF-16
[23:43] <MisterN> they are well-defined iff all characters are on the BMP
[23:48] <Dantman> Anonymous classes! http://pastie.org/858813

 

 

Logs by date :