Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2010-10-16:

[1:23] <Wes-> whatever happened to the CommonJS logo we had designed for us last year?
[1:23] <Wes-> It doesn't even come up when you google CommonJS
[1:24] <Wes-> but Kris Kowal, dressed in a witch's costume, does
[1:35] <deanlandolt> ha
[1:35] <deanlandolt> i can dig it up...i really liked that
[1:45] <Wes-> Yeah, I thought it was pretty classy
[1:45] <Wes-> I think it was Hannes' wife that drew it?
[1:45] <Wes-> Irakli should be hanging out in here
[3:55] <deanlandolt> Wes so no require.main in gpsee?
[4:02] <deanlandolt> no assert module either?
[11:30] <Wes-> deanlandolt: No require.main - what's that for?
[11:31] <Wes-> deanlandolt: No assert module either, haven't needed one - trivial to add though. I suspect Narwhal's would drop right in.
[11:32] <Wes-> deanlandolt: My testing infrastructure kind of sucks right now, I really need to improve it
[11:34] <Wes-> deanlandolt: Argh, maybe the narwhal one doesn't drop right in, util.js throws a bajillion strict-mode warnings
[11:36] <Wes-> deanlandolt: Something worth mentioning, gsr -S turns off strict mode. GPSEE's strict mode is not ES5 "use strict", but it's similar and predates it. Unification should come this quarter, maybe next.
[11:37] <Wes-> The warning thrown, FWIW, is trying to change the length of a function (operator)
[11:45] <deanlandolt> yeah, no big deal
[11:45] <deanlandolt> yeah, i'm familiar w/ sm's strict mode
[11:46] <Wes-> deanlandolt: Aside from that, assert.js seems to run on GPSEE. Now, test.js is another thing entirely.
[11:46] <deanlandolt> Wes- home come require("binary").ByteString("foo") doesn't work?
[11:46] <deanlandolt> not worried about test, just doing simple assertions
[11:46] <Wes-> should it?
[11:46] * Wes- doesn't remember
[11:46] <deanlandolt> how do you create a ByteString or a ByteArray?
[11:46] <deanlandolt> ByteArray([97,98,99]) don't work either
[11:47] <Wes-> !!
[11:47] <Wes-> what happens?
[11:47] <deanlandolt> could be related to this warning i'm getting: the mozshell not found thing
[11:47] <Wes-> deanlandolt: Wait, did you forget 'new' ?
[11:47] <deanlandolt> but it's saying gpsee.module.ca.page.binary.ByteString.cast.type: String objects are no castable to gpse.module.ca.page.binary.BiteString
[11:48] <Wes-> deanlandolt: you forgot 'new'
[11:48] <Wes-> "mozshell not found" - I thought I excised that a year ago
[11:48] <Wes-> Hey, are you running the jsie REPL?
[11:48] <deanlandolt> yeah, the jsie repl's pretty nice :)
[11:48] <deanlandolt> but in any event, i get the same error w/ new require("binary").ByteString('abc')
[11:49] <Wes-> deanlandolt: Heh - that hasn't been tested in a LONG time, glad that's still working. I plan to improve it at some point. "readline()" is 100% JS
[11:49] <Wes-> (using a curses library)
[11:50] <Wes-> deanlandolt: new (require("binary").ByteString)("abc") would be the correct syntax
[11:50] <deanlandolt> ugh...shite
[11:50] <Wes-> deanlandolt: JS parses that statement as "ByteString('abc')" off the require("binary") constructor
[11:50] <deanlandolt> how could i have missed that :)
[11:51] <Wes-> don't feel bad, it happens to me regularly. the new-require idiom seems natural, but sucks
[11:55] <deanlandolt> i usually just avoid new completely and play the if (!this instanceof Constructor) return new Constructor(args)) trick
[11:57] <Wes-> deanlandolt: In GPSEE we actually a formal meaning for new-less "constructor" functions that return Binary types -- they are "casters".... So, new a=new Memory(1000); call_c_function(a); bs=ByteString(a); ba=ByteArray(a)
[11:57] <Wes-> bs and ba both have the backing store, or a copy of it
[11:57] <deanlandolt> ah, fair enough
[11:57] <Wes-> When I get copy-on-write done, there will be no copy
[11:59] <Wes-> deanlandolt: If a new-less standard becomes, well, standard, I may have to get more careful with my casting functions. It's possible, from the C side, to detect which class a given native object is - making it possible to implement your basic polymorphic function
[12:00] <deanlandolt> well at least w/ the stuff i'm writing it shouldn't have any impact on you
[12:00] <deanlandolt> i just exporter a Constructor fn that defers to the platform
[12:01] <deanlandolt> (in gpsee new ByteArray i guess)
[12:02] <deanlandolt> Wes- what's the deal w/ the shebang warning? i mean, if i'm running the same script in gpsee, node, narwhal, rhino and flusspferd how am i supposed to shebang that?
[12:03] <Wes-> deanlandolt: Do node and narwhal support a shebang mode?
[12:03] <deanlandolt> node does
[12:03] <deanlandolt> (IIRC)
[12:04] <Wes-> deanlandolt: What flusspferd and gpsee do for interopt is a #! /usr/bin/env commonjs she-bang line
[12:04] <Wes-> deanlandolt: then you need to linke commonjs to your preferred interpreter
[12:04] <deanlandolt> gotcha...
[12:04] <Wes-> deanlandolt: custom she-bang options (i.e. gsr -dd) can then be specified in leading comments, // gpsee: -dd
[12:04] <Wes-> I think flusspferd and v8cgi support this too
[12:05] <Wes-> deanlandolt: OTOH - if you just want to run gsr -f program.js and to skip the shebang, use -F
[12:05] <deanlandolt> i forgot about v8cgi...have to add that to the list
[12:05] <Wes-> The warning is a more polite version of "#! /usr/bin/gsr: syntax error" :)
[12:05] <Wes-> FWIW, there exists gsr -h
[12:06] <deanlandolt> i bet node would take a patch for the shebang options
[12:06] <deanlandolt> that's pretty sexy actually
[12:07] <Wes-> deanlandolt: I wracked my brain trying to come up with that, there was interest in the three implementers but the community at the time (narwhal was the dominant impl)
[12:07] <Wes-> s/but/but not/
[12:07] <deanlandolt> so what does -dd do?
[12:07] <Wes-> Ah, there's the bug on that: http://code.google.com/p/gpsee/issues/detail?id=12&can=1&q=shebang
[12:08] <Wes-> deanlandolt: each d enables one more level of verbosity. 3 is very verbose
[12:08] <deanlandolt> oh, okay
[12:27] <Wes-> deanlandolt: what's your email address?
[12:28] <deanlandolt> dean@deanlandolt.com
[12:30] <Wes-> hannesw_: Does your CommonJS platform support launching programs via #! ?
[12:31] <hannesw_> Wes-: yes it does
[12:31] <Wes-> hannesw_: Did we ever discuss comment-embedded options?
[12:31] <hannesw_> oh yeah
[12:31] <Wes-> Did you implement?
[12:31] <hannesw_> we even have an issue for it
[12:32] <Wes-> RingoJS, right? What's your email, I'm going to copy you on this
[12:32] <hannesw_> nope :)
[12:32] <Wes-> Which one? I can never keep track of these things!
[12:32] <hannesw_> http://github.com/ringo/ringojs/issues#issue/43
[12:32] <hannesw_> hannes at helma dot at
[12:32] <Wes-> Thanks
[12:42] <Wes-> deanlandolt: Okay, sent ryan a note suggesting he get on board with my pet project :)
[12:43] <deanlandolt> nice
[12:44] <Wes-> I just (mentally) came up with a trivial way to implement script-supplied event loops in GPSEE
[12:44] <Wes-> holy moly
[12:49] <deanlandolt> Wes- so re: require.main...how do you test if you're module has been run directly in gpsee?
[12:49] <deanlandolt> (so that you can execute your tests)
[12:50] <Wes-> deanlandolt: that's a good question. If you know the name of the test driver, require("program") will give you it's exports
[12:50] <Wes-> deanlandolt: Can you show me the code?
[12:51] <deanlandolt> it's not a separate test file...i havea shim.js file that just detects your engine and loads the appropriate shim...
[12:51] <deanlandolt> then there'sj ust a test fn that gets run if it's been invoked directly
[12:52] <deanlandolt> i'm doing this temporarily but it's incredibly hacky:
[12:52] <deanlandolt> function isMain() {
[12:52] <deanlandolt> // HACK for gpsee
[12:53] <deanlandolt> if (!require.main) return module.id && !!module.id.match(/commonjs\/binary\/shim/);
[12:53] <deanlandolt> return require.main === module
[12:53] <deanlandolt> }
[12:53] <deanlandolt> "utf-8": "utf8"
[12:53] <Wes-> Hm, so the spec says that we MAY supply require.main ;) -- and that require.main returns "module" from the program module
[12:53] <deanlandolt> err...shit...sorry, copy/paste error
[12:53] <Wes-> heh
[12:54] <deanlandolt> yeah, narwhal and node do it...i think ringo too
[12:54] <deanlandolt> not sure about the others
[12:54] <deanlandolt> but it's useful
[12:54] <Wes-> This might be really easy to add, I keep the program module's handle around already on the C side, let's see how hard it is to add
[13:02] <Wes-> hey, I just realized require.main is neat
[13:02] <deanlandolt> :)
[13:02] <Wes-> require.main.require from any module should let you require any resource as though you were the top-level program
[13:02] <deanlandolt> oh? heh, that is neat
[13:03] <Wes-> ASSUMING that all requires aren't implemented with the same function
[13:03] * deanlandolt is trying to think of a cap leak
[13:03] <Wes-> which isn't true by spec, but seems to be true by impl
[13:03] <Wes-> or, no, wait
[13:03] <Wes-> Never mind, require.main links to the module, not the global
[13:03] <Wes-> require("system").global on the other hand..
[13:04] <deanlandolt> ah yes...the module free var
[13:04] <deanlandolt> is the system module standard?
[13:04] <deanlandolt> heh...crap, the scope of my little binary project has already creeped into reinventing narwhal w/o the engines stuff :)
[13:11] <Wes-> I have an a simple engine.js if you need it
[13:11] <deanlandolt> engine.js?
[13:11] <Wes-> yeah, from when I was trying to get narwhal-lib running
[13:11] <deanlandolt> oh, cool...yeah
[13:13] <Wes-> http://www.page.ca/~wes/engine.js
[13:13] <Wes-> http://www.page.ca/~wes/os-engine.js
[13:13] <Wes-> neither well tested ;)
[13:15] <deanlandolt> nice
[13:18] <Wes-> the first one demonstrates nicely how GPSEE can call into C with a fairly JS-y feel (I paid a lot of attention to type coercion and what not, rather than opting for strict typing)
[13:39] <Wes-> deanlandolt: hg pull && hg up && make build && sudo make install
[13:40] <Wes-> require.main, lightly tested
[13:40] <deanlandolt> awesome...testing now
[13:42] <deanlandolt> success :)
[13:44] <Wes-> beauty, eh? Once I figured out what to juggle it was about a 12 line patch
[13:44] <deanlandolt> thanks Wes-
[13:44] <Wes-> n/s
[13:44] <deanlandolt> now i have to figure out why the asserts are all passing :-/
[13:59] <deanlandolt> d'oh...ringo doesn't support require.main :-/
[14:04] <Wes-> deanlandolt: rofl
[14:04] <Wes-> deanlandolt: what about
[14:04] <Wes-> if (!require.main) require.main = module
[14:04] <Wes-> in your program?
[14:05] <deanlandolt> aye, nice
[14:05] <Wes-> that will lead to either require.main being the program module's module free var or undefined in non-program modules
[14:06] <Wes-> depending on the underlying implementation of require
[14:06] <Wes-> (i.e. whether each module gets it's own copy or it's resolved up the scope chain)
[14:06] <deanlandolt> yeah...just have to do it at the top
[14:06] <Wes-> of the program module, not the others
[14:06] <Wes-> then the others, you just do if (require.main !== module) as your test
[14:06] <Wes-> it will pass either way
[14:07] <deanlandolt> yeah, but there may be modules i require that also play the if not require.main trick
[14:07] <deanlandolt> so i just have to be sure i do that before any requries
[14:07] <deanlandolt> requires
[14:20] <deanlandolt> Wes- is there a good feature test to tell for sure you're in gpsee?
[14:21] <deanlandolt> like some special global you do or some key somewhere?
[14:21] <Wes-> deanlandolt: require("gpsee") is very fast on gpsee (no I/O) and throws everywhere else
[14:21] <deanlandolt> k
[14:21] <Wes-> or, gpseeNamespace is a string
[14:23] <deanlandolt> even better, no try...perfect
[14:27] <deanlandolt> hmm...it appears ringo defines main on require, but for some reason i'm not seeing it...odd
[14:30] <Wes-> typeof require.main ?
[14:34] <deanlandolt> "undefined"
[14:35] <deanlandolt> i see it in the source, and i built from source...but oh well, i'll take this up w/ hannes when he's back -- your if(!require.main) boilerplate should do the trick
[14:38] <Wes-> wierd, is it defined in non-main modules?
[14:39] <Wes-> I had to juggle GPSEE's module loader so that 'module' was visible before I tried assigning to require.main when handling the program module
[14:39] <deanlandolt> its defined in java when they put require into the RingoGlobal context...
[14:39] <deanlandolt> but i'm sure it's just something goofy
[14:40] <deanlandolt> perhaps an old version of the rhino jar's being used so defineProperty falls flat
[15:12] <Wes-> deanlandolt: Hey, Dean - do you understand JSGI?
[15:12] <deanlandolt> heh, yeah
[15:12] <Wes-> deanlandolt: Is there much to it? Do test suites exist? Do "real" applications use JSGI?
[15:13] <deanlandolt> yeah, all my real applications do...pintura's a pretty huge stack that uses it...
[15:13] <Wes-> I've been thinking about adding better webserver support here, but aside from knowing what *I* want, I have no idea what "the market" wants (besides node)
[15:13] <deanlandolt> the spec itself needs to be hardened (that's actually the whole reason for my whole binary endeavor, and my next "streams" endeavor)
[15:13] <Wes-> And JSGI does not rely on the webserver being a giant event loop, right?
[15:13] <deanlandolt> no, doesn't require it
[15:13] <Wes-> (I want something that sits inside apache cleanly)
[15:14] <deanlandolt> ah, well then yeah, JSGI's about as clean as you can get
[15:14] <deanlandolt> requests have a well defined set of keys and a body you can forEach over to get your byte strings...
[15:14] <Wes-> deanlandolt: Interesting, maybe I should give you an apache + gpsee + jsgi stack someday to help your spec-hardening thinking :)
[15:14] <Wes-> deanlandolt: Do you know if any of the JSGI implementations are mostly-JS?
[15:15] <deanlandolt> and responses have a status key (int), headers key (object of lowercase header keys and ascii string values) and a body that looks the same (forEach stream)
[15:15] <deanlandolt> yeah, most of them are...
[15:15] <deanlandolt> here's "most" of the jsgi impl...
[15:15] <Wes-> that's actually interesting, yeah, and helping "memory" refresh
[15:15] <Wes-> What I want to do is reflect apache internals into script
[15:15] <Wes-> and then write JSGI with that reflection
[15:16] <deanlandolt> http://github.com/280north/jack/tree/master/lib/jack/handler/
[15:16] <deanlandolt> ashb did one called juice...and kriszyp did one on node called jsgi-node
[15:17] <deanlandolt> and ringo has an impl
[15:17] <Wes-> Oh wow, that runs on v8cgi it should run on GPSEE-cgi without a lot of thought
[15:17] <deanlandolt> yeah, v8cgi implements jsgi
[15:17] <Wes-> Yeah, I remember ashb's juice, I was thinking that it relied on a Flusspferd web server
[15:17] <deanlandolt> i mean, it's insanely easy
[15:18] <Wes-> Is there a "hello world" app in there somewhere?
[15:18] <deanlandolt> yeah...here's the hello world:
[15:18] <deanlandolt> http://github.com/280north/jack/blob/master/examples/example.js
[15:19] <deanlandolt> that's a little more complicated than necessary actually...
[15:19] <deanlandolt> and that's actually an older version of jsgi :-/
[15:19] <deanlandolt> (we've gone with lowercase keys now)
[15:19] <deanlandolt> but hello world is really as easy as this:
[15:19] <Wes-> So, if I was doing that under apache - I would copy example.js into, say, /var/www/jsgi, and rig apache up so that .js files in that directory were handled by gpsee running a JSGI module?
[15:19] <deanlandolt> function() {
[15:19] <deanlandolt> return function(request) {
[15:20] <deanlandolt> return { status: 200, header: {}, body: ["hello world"] }
[15:20] <deanlandolt> }
[15:20] <deanlandolt> }
[15:20] <Wes-> Is the jack on 280north a new-enough JSGI to run, say, pintura?
[15:20] <deanlandolt> probably, yeah, and then eventually try to roll into an fcgi setup
[15:21] <Wes-> wow, that's pretty straight forward indeed
[15:21] <deanlandolt> no (i'm been remiss in updating)...check out the kriszyp/jack repo
[15:21] <Wes-> Does pintura (and other jsgi apps?) rely on things like persistent free variables from one request to another?
[15:21] <deanlandolt> usually the top function would be "function(nextApp) {" ...so you can delegate to the next app in the chain (mw) but at your endpoint application you have nothing to delegate to so i left it out
[15:21] <deanlandolt> no, never
[15:21] <Wes-> or is the fcgi suggestion a perf suggestion only?
[15:21] <deanlandolt> perf only
[15:22] <Wes-> Cool
[15:22] <deanlandolt> everything exists in the request context, and everything returns is in the response object
[15:23] <deanlandolt> you can do all kinds of nice optimizations like return fs.open('file'); if you update your fs.open to put a "path" key on your file object...then you don't even have to actually open your file, in your jsgi handler you can catch the fact that there's a path key on the response and just set the x-sendfile header (in nginx...does apache have something like that?)
[15:24] <Wes-> That means a proof-of-concept with CGI should work fine. And GPSEE's about as fast as CGI can get -- typical time overhead on a slow sun box is about 4ms per request
[15:24] <deanlandolt> what's really cool is if some mw decides it wants to look at the body it won't screw anything up...it will just forEach over the provided "read" function to get the body and replace it...thus overwriting the path key
[15:24] <deanlandolt> heh, yeah
[15:24] <deanlandolt> that's pretty much minimum process opening time isn't it?
[15:24] <Wes-> (we even cache compiled byte code if the perms in the dir allow)
[15:24] <Wes-> deanlandolt: Pretty much, yeah
[15:24] <deanlandolt> yeah, i saw that...i'm gonna have to set up a .gitignore :)
[15:25] <deanlandolt> but yeah, jsgi has some pretty awesome ideas that went into it...there were a ton of really good discussions on the list and some awesome symmetry shook out
[15:25] <Wes-> Since we use GPSEE in a systems programming environment around the UNIX model, we have spent some perf time on launch + shutdown. That's about all that we've optimized, though. I believe in measure-first, and haven't needed to measure yet. :)
[15:26] <Wes-> deanlandolt: -C flag will turn the cache files off, FWIW
[15:26] <Wes-> Actually, maybe even // gpsee: -C in your code, but I haven't tested that
[15:26] <deanlandolt> cool...i'll give it a shot
[15:26] <Wes-> s/in/at the very top of
[15:26] <deanlandolt> yeah, i have the shebang in but i haven't linked up commonjs to anything so i'm just invoking gpsee directly
[15:27] <deanlandolt> i really dig the shebang idea though
[15:27] <Wes-> Drat, the -C doesn't work in the // gpsee: inline comment due to a timing issue (we've already created the cache before we see those comments)
[15:28] <deanlandolt> ah, oh well
[15:28] <ashb> Wes-: juice is (or sohuld be) pure jsgi 0.3
[15:28] <deanlandolt> yeah iirc
[15:28] <ashb> but i've done nothing on it since Jan or something :(
[15:28] <Wes-> ashb: So, juice is your framework that lives on top of JSGI?
[15:28] <ashb> Wes-: yeah
[15:29] <Wes-> ashb: What's your JSGI written in?
[15:29] <ashb> that's Zest - boost::asio + flusspferd
[15:29] <deanlandolt> http://github.com/ashb/Zest
[15:30] <deanlandolt> Wes- if you're looking for a clear guide i can upgrade jack's cgi handler
[15:30] <deanlandolt> or you could just do it however you want and i can come in behind you
[15:31] <Wes-> deanlandolt: I was going to try and port v8cgi.js to work with GPSEE
[15:31] <deanlandolt> oh yeah, that's about the same
[15:31] <Wes-> deanlandolt: Oh, wait - does JSGI need to be able to read the stdin stream directly (post data?)
[15:31] <deanlandolt> well, the handler needs to provider a body forEach fn that lets you loop over that...
[15:31] <ashb> request body should be a stream in env.jsgi.input
[15:31] <ashb> or similar, yes
[15:31] <deanlandolt> you could just hand the stdin dir
[15:32] <deanlandolt> directly
[15:32] <Wes-> I have a CGI module that turns GET and POST data into an object that's vaguely like PHP's $_GET[] -- but I think I'll skip that for JSGI and go straight for stdin
[15:32] <deanlandolt> yeah, you can do that in mw
[15:32] <deanlandolt> middleware
[15:32] * Wes- nods
[15:33] <deanlandolt> it's nice to get the raw request...as close to http as you'd want if you were sane...then just enhance it through a chain
[15:33] <Wes-> require("cgi") also handles file upload and stuff, but I'm not real happy with the file upload handling, other than it works :)
[15:33] <ashb> if the way of handling multiple params wasn't so varied and contentions i'd say it makes sense in JSGI
[15:33] <Wes-> Yeah, that's the good go-forward approach
[15:33] <deanlandolt> yeah, again, there's middleware to handle all that too...
[15:33] <ashb> but foo=bar&foo=baz vs foo[]=bar&foo[]=baz etc
[15:33] <Wes-> You know, I should write a CommonJS port of my PHPSession class
[15:33] <Wes-> It lets me read PHP Session files, making migration a snap
[15:33] <deanlandolt> check out jack -- there's a ton of stuff there...but it's all screwy because we don't ahve a standard fs/binary/stream world...that's what i'm trying to fix :)
[15:34] <Wes-> deanlandolt: FWIW - wrt streams - #1 most useful "thing" is the ability to read a file into an ByteArray or an array of String lines
[15:34] <deanlandolt> yeah, no doubt
[15:35] <Wes-> deanlandolt: That lets a) users do about 80% of "real work" (like reading config files) while b) giving engine authors a LOT of knowledge so we can optimize the snot out of those
[15:35] <Wes-> deanlandolt: Concretely, in GPSEE, if you read the whole file, we memory-map it, which takes no I/O at all if it's been recently read
[15:35] <deanlandolt> yeah, but the `read(N)` paradigm isn't flexible enough
[15:35] <Wes-> (by any other process)
[15:36] <Wes-> deanlandolt: No reason why require("fs-base").open().readAll() can't return a giant ByteString
[15:36] <Wes-> or maybe "snapshot" instead of readAll
[15:36] <deanlandolt> you /MUST/ use sync to do that
[15:37] <ashb> sync or risk blowing your stack
[15:37] <ashb> (by using a re-entrant event loop)
[15:38] <Wes-> deanlandolt: You can always pass a continuation-style call back when you are in async-land to get the same perf benefit
[15:38] <deanlandolt> well, you can get away w/o a reentrant event loop if you return a promise
[15:38] <deanlandolt> that resolves to the whole file...
[15:39] <deanlandolt> and we can support that
[15:39] <Wes-> deanlandolt: The important thing here isn't the calling convention, it's that you give the engine the opportunity to read the file the fastest way possible -- if that's what you need
[15:39] <deanlandolt> forEachableToString has been written a dozen times :)
[15:39] <deanlandolt> yeah, that's a good point
[15:39] <deanlandolt> i suppose the engine could provide their own forEachableToString (or readAll, or whatever) stream helper method
[15:40] <deanlandolt> and just do what it does
[15:40] <deanlandolt> and if it's already memmapped not even return the promise, just return directly
[15:40] <deanlandolt> well, actually, that's how you'd do it w/ sync io anyway, cached results or not
[15:40] <deanlandolt> it would just return a lot faster
[15:41] <deanlandolt> but Wes- the important thing is that JSGI doesn't /require/ you to use promises anywhere...you just /can/ use them in lieu of typed return values if your async
[15:42] <deanlandolt> thus, if you want to write interoperable middlweare you have to use things like promise.when(someReturnedValueThatCouldBeAPromise, function(result) { /*success*/ }, function(e) { /*fail*/ });
[15:42] <Wes-> *nod* - the interesting case for mmap()d files, IME, isn't stuff like JSGI streams, though -- it's silly things, like configuration files, server-side session data and so on
[15:43] <Wes-> So, for example, one of our local products stores state something like PHP session files, except JSON
[15:43] <Wes-> we read those state files in with mmap on JSON.parse() them
[15:43] <Wes-> Using mmap() gives the kernel a good clue to keep them pinned in RAM, which lowers our request handling time
[15:44] <Wes-> And we can give the kernel the mmap() clue by always reading the entire file in a single operation
[15:44] <Wes-> (there are other ways, of course -- but that one's really easy)
[15:45] <deanlandolt> yeah, but in the streaming case i'm talking about the engine could easily provide their own readAll helper that does what it does...
[15:45] <deanlandolt> it wouldn't have to use the underlying forEach stream, but the key is that it's there
[15:46] <deanlandolt> i'm trying to figure out how to make that feasible with a stream spec
[15:46] <deanlandolt> well, it's already /feasible/ but if it were explicit than the engine could
[15:46] <deanlandolt> ...err, premature send...the engine could provide all the requisite helpers it wants and optimize the shit of them
[15:48] <Wes-> Oh, yeah, of course - readAll could *totally* be implemented other ways. The key for good API design in this case, I think, is identifying horribly common use cases (like reading the whole file) and poking an API that tells the back end about the use-case.
[15:49] <Wes-> So, for example, with no readAll in the spec, I have to either write an engine extension to get the same option, or observe the fact that the users *seems* to be in a readAll loop, or just always read All and hope that's okay
[15:49] <Wes-> (which it might not be with a multi-gigabyte log file)
[15:50] <deanlandolt> yeah, but what do you do if a user readAll's on a multigigabyte file?
[15:50] <deanlandolt> set a system-defined (customizable) limit and throw?
[15:50] <Wes-> deanlandolt: Exactly what he asked us to do. Either readAll or throw
[15:50] <Wes-> We will throw on OOM
[15:50] <Wes-> And it's uncatchable
[15:50] <deanlandolt> okay
[15:50] <deanlandolt> well, but you can detect it and throw before OOM
[15:50] <Wes-> Or, actually, it might be catchable
[15:50] <Wes-> I think the mmap will fail, yeah, and that will cause readAll to throw
[15:51] <Wes-> But the mmap will only fail if there isn't enough free address space
[15:51] <Wes-> which acutally means that readAll() can handle bigger files than piecemeal read-assemble
[15:51] <deanlandolt> but yeah, i agree about that...w/o defining the use cases we're sol
[15:51] <Wes-> this is because mmap()'d fiels become part of the process's address space, and are not limited by VM allocation
[15:52] <deanlandolt> still, there's a hard limit
[15:52] <Wes-> Yep - address space
[15:52] <Wes-> pointer size
[15:52] <Wes-> or less, depending on arch
[15:52] <Wes-> practical limit on most machines is about 2^40
[15:52] <deanlandolt> and you shouldn't be piecemeal read-assembling on files you don't KNOW are small...it's a code smell
[15:52] <Wes-> deanlandolt: agree
[15:53] <deanlandolt> but yeah, i'll keep that in mind
[15:53] <Wes-> Lots of funny corners you get into when you have lots of different environments for your modules :)
[15:53] <Wes-> I'm pretty sure there aren't many systems guys today, but I also bet there will be in 2 years
[15:54] <Wes-> GPSEE beats the hell out of shell scripts for many, many applications
[15:57] <deanlandolt> yeah i bet it does
[15:57] <deanlandolt> shell scripts are just painful
[16:01] <Wes-> And slow!
[17:53] <deanlandolt> Wes- heh...so i just got back to the binary stuff...i'd assumed there was something wrong with the assert fn because nothing was failing...turns out it just works...sweet :)
[17:54] <deanlandolt> (i'm not testing all that much, of course)
[18:00] <deanlandolt> oh, umm, nvm
[18:01] <deanlandolt> can you think of any reason why gpsee would just completely drop mid execution, right in the middle of a function? (happens somehow during the second call to assert.equal)
[19:39] <deanlandolt> ashb: there's no Object.create in flusspferd?
[19:45] <ashb> should come from spidermonkey?
[19:45] <ashb> we might not have got round to bootstrapping it up
[19:51] <deanlandolt> ah well, i just stuck an es5 shim on it and it's fine
[19:58] <deanlandolt> sweet...the tests pass in gpsee, narwhal and node...i still have to override a few signatures for flusspferd and ringo
[19:58] <ashb> tests for?
[19:58] <deanlandolt> ashb: Wes convinced me to do up a package with the simplistic binary stuff i proposed
[19:58] <ashb> ah
[19:59] <deanlandolt> i'm going to carve in kriszyp's promised-io/fs stuff (and do sync, promised, and callback style apis)
[20:00] <deanlandolt> and get a stream spec together that illuminates the differences (and parody) of the three primary stream paradigms (evented, forEachable and read(n)
[20:01] <deanlandolt> the goal is a simple, light weight module that node folks can include w/o changing their beloved apis taht will allow all of us to use all the work that's going into node-specific packages
[20:02] <deanlandolt> well, that's one goal...i'd also like to make it so that we can shim the node api to use node packages w/o the author using the commonjs apis
[20:23] <deanlandolt> ashb or Wes: i can't find an api to just add a byte to a bytearray at an index...the best i can do is splice...
[20:23] <deanlandolt> is that right? is there something better?
[20:23] <ashb> a[1] = 2
[20:24] <ashb> perhaps
[20:24] <deanlandolt> does that work in flusspferd?
[20:24] <deanlandolt> it fails in narwhal but i can do the splice hack for narwhal
[20:24] <ashb> not sure tbh
[20:24] <ashb> been a while :/
[20:25] <deanlandolt> cool...i'll feel it out
[20:42] <deanlandolt> ringo passes now :)
[23:18] <Wes-> deanlandolt: Yes I can think of a reason. That is typical of a specific type of programming error that can be made in spidermonkey -- basically, you can (from C) throw without setting an exception. I actually have plans to detect those at some point.
[23:18] <Wes-> deanlandolt: Do you have a rough idea what function is doing that? It's probably a 2-second fix once the function is identified
[23:20] <Wes-> deanlandolt: BTW, if I'm not around, hdon is also extremely gpsee.knowledgeable
[23:25] <Wes-> deanlandolt: FWIW -- I'm doing a quick survey of the binary module to try and avoid some of those cases. It was written when both hdon and I were getting our spidermonkey legs (and binary/b is an incredibly difficult spec to implement in C)
[23:41] <Wes-> deanlandolt: Well, I found one such possible case thus far -- if you call .charAt or .byteAt on ByteArray.prototype -- I doubt you are, though
[23:41] * Wes- fixes

 

 

Logs by date :