Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2010-04-20:

[0:00] <jbrantly> jhuni: another question is how is the data specified. For instance, ashb doesn't like the concept of analyzing code for it, but from my perspective, having it in a separate file is a PITA for browser loaders. So stuff like that needs to be figured out. Perhaps some of the metadata stuff could be added to a new Transport format to solve the browser issue.
[0:01] <jhuni> jbrantly: package managers can separate it, that is under the usage section
[0:02] <jhuni> jbrantly: hmm, perhaps commonjs scripts that use require could have a script try to extrapolate their metadata into a JSON file?
[0:03] <jhuni> jbrantly: at least for people that don't explicity write metadata
[0:04] <jhuni> jbrantly: another use is you could have a mixins directive which mixins another JSON metadata into the current one, that way if you have a list of requirements you currently use you could make them into a mixin
[0:05] <jhuni> jbrantly: also that directive could be used as such "mixins": ["Modules/1.1.1"] to mixin all the stuff that handles modules 1.1.1
[0:05] <jbrantly> jhuni: my thought is more the other way around. The metadata could help determine dependencies. If a module doesn't specify metadata, so be it.
[0:06] <jhuni> jbrantly: if you don't have any metadata a script executes as is
[0:06] <jhuni> jbrantly: so I suppose you could say it is merely a helper
[0:07] <jbrantly> jhuni: sure, but you just mentioned that tools could be used to try and create metadata. I'm saying in my mind its the other way around. The metadata is supposed to help the tools.
[0:07] <jhuni> jbrantly: the metadata would just be a standard storage mechanism for scripts that can determine dependencies in that case
[0:08] <jhuni> jbrantly: you are getting the metadata for the script anyways...
[0:09] <jhuni> jbrantly: tools can be built around the metadata format rather then being built around handling the error-prone require process
[0:09] <jbrantly> exactly
[0:09] <jbrantly> but the metadata *has* to be optional
[0:09] <jhuni> jbrantly: all I ask for then is a way to determine if a script is commonjs or not
[0:10] <jbrantly> jhuni: I still don't get that. This is a CommonJS specification. Of course its a CommonJS module
[0:10] <jhuni> jbrantly: ok then the question is how to determine if the module is using Modules/1.1.1
[0:11] <jbrantly> jhuni: one thing that *may* be useful is the version of modules
[0:11] <jbrantly> 1.0 or 1.1.1 for example, and to not supply the "module" free variable for 1.0 modules as something that the loader could do differently
[0:20] <jhuni> jbrantly: btw how do you specify what builtins you want to use, like can you do require('Array.prototype.reduceRight')
[0:21] <jbrantly> jhuni: err. I don't know if thats necessary. You can certainly stick it into your proposal, but I would probably vote against it.
[0:22] <jbrantly> jhuni: btw, I'm glad we could talk here. Much faster than the ML for getting some of this sorted out :)
[0:22] <jhuni> jbrantly: its necessary for some high performance applications that also want to be cross-browser
[0:22] <jhuni> jbrantly: this metadata format is mainly to make web browsers happy =/
[0:25] <jhuni> jbrantly: the point is you can specify Array.prototype.reduce so that you can load less files for Firefox 3.5 well still working exactly the same in IE8, which is advantageous
[0:25] <jhuni> jbrantly: the basic point is the metadata for builtins could be useful
[0:26] <jbrantly> jhuni: then include it and talk about why :)
[0:27] <jhuni> jbrantly: also so that loaders like Yabble don't have to do a bunch of ActiveXObject stuff to make old versions of IE happy
[0:28] <jhuni> jbrantly: By that I mean you don't have to load the ActiveXObject stuff if you are using Firefox 3.6 but you will load the appropiate stuff for IE6
[0:28] <jhuni> jbrantly: and the appropriate java-based normalizer for opera 7 as well
[0:30] <jhuni> jbrantly: I removed the stuff about the Use Server
[0:31] <jhuni> jbrantly: now all it contains is the overview, the format, example codes, and very brief stuff about usage
[0:37] <jhuni> jbrantly: also I noticed yabble could do with Array.prototype.indexOf and Array.prototype.filter, this would make it smaller in Firefox
[0:39] <jbrantly> does anybody have an objection with me adding Yabble as an implementation to the wiki?
[0:40] <jhuni> don't you think yabble would be better w/ dependencies?
[0:40] <jhuni> It would certainly be able to be smaller over HTTP
[0:41] <jbrantly> jhuni: maybe. The idea was to be self-contained. There's a decent chance that there *are* no other dependencies since Yabble is loading everything.
[0:42] <jhuni> jbrantly: I had the same idea with the OpenJSAN loader but now I think Metadata is the way to go
[0:44] <jhuni> jbrantly: every script that is being sent over HTTP should have dependencies, that way you just load the files you need, the metadata system is built to be efficient in the client-side environment
[0:46] <jbrantly> jhuni: I don't like the idea of a "metadata system". Then the "metadata system" is somewhat taking the place of the loader. Whats the point?
[0:46] <jhuni> jbrantly: what term would you prefer?
[0:46] <jhuni> jbrantly: the point is to make high performance client side websites
[0:47] <jhuni> jbrantly: to have a minimum of http requests and file size
[0:48] <jbrantly> jhuni: thats not my point :) my point is to make a simple metadata proposal.
[0:50] <jhuni> jbrantly: I am cleaning the proposal up, so it will be more simple and focused
[0:51] <jbrantly> jhuni: good. But it seems that you're still looking to have something that is essentially lower-level than the loader itself, and I think that is probably out of scope.
[0:51] <jbrantly> jhuni: you're basically making the "metadata system" a loader in and of itself
[0:52] <jbrantly> you have to load code in order to load the loader
[0:52] <jhuni> there is code working on the server-side
[0:53] <jhuni> you don't need to load code to load the loader, the server-side handles the metadata of the loader...
[0:53] <jbrantly> jhuni: gotcha. And that is fine. I'm not saying it shouldn't be in the proposal. I'm just saying I don't think Yabble would use it, for example/
[0:53] <jhuni> the loader can be the first thing you load onto your page, or if you want another script can be
[0:53] <jhuni> well then yabble probably wouldn't be what a lot of websites use
[0:54] <jhuni> cause I think they would prefer a solution that minimizes file size
[0:54] <jbrantly> well there are almost no websites using CommonJS today, so I don't have to live up to high standards :D
[0:54] <jhuni> or perhaps you can make two versions of yabble.
[0:54] <jbrantly> jhuni: Yabble is ~2K minimized and gzipped. I think it's small enough.
[0:54] <jhuni> or you can have a version of Yabble with the META thing and another version that already has all its dependencies loaded for it so that it doesn't require any META stuff
[0:55] <jhuni> one of the options in the META thing is you can do the handling and remove the META thing
[0:56] <jhuni> for example, you can list builtins like [XMLHttpRequest], and then the handler can build a version of Yabble with something like if (!XMLHttpRequest) {}
[0:56] <jhuni> that way users without any META system can run yabble normally
[0:56] <jhuni> actually any script that is using metadata has that option
[0:57] <jhuni> you can pack a version of yabble that has its metadata already handled and it can be packed still, and when it is used by some server-side users the server can use the metadata to improve performance
[0:58] <jhuni> jbrantly: is that the sort of thing you are suggesting then? Because you will have the option to pack scripts without any reference to the META function
[0:58] <jhuni> jbrantly: every script will have that option
[1:01] <jbrantly> I'm not suggesting anything other than I personally don't think its very interesting/useful, but perhaps others will
[1:02] <jhuni> jbrantly: why should you have to have code that deals with normalizing XMLHttpRequest in a loader script??
[1:02] <jhuni> they are like two different things, a good practice would be to separate them
[1:03] <jbrantly> jhuni: because the loader script is just some static script that will get loaded for all browsers. Or it will get packaged up with all of the modules (but first), again for all browsers.
[1:04] <jhuni> jbrantly: that doesn't really explain why the loader script should handle XMLHttpRequest normalization...
[1:04] <jhuni> jbrantly: but yeah you can just handle the metadata well packing the script into its loadable form
[1:05] <jhuni> jbrantly: btw the META.js script has metadata...
[1:05] <jbrantly> jhuni: thats a cool idea and all, but it just doesn't really belong in the CommonJS specification I don't think.
[1:06] <jhuni> jbrantly: where does it belong?
[1:06] <jbrantly> in your own custom browser loader implementation, I guess. One that requires some kind of server support.
[1:07] <jhuni> jbrantly: ...
[1:07] <jhuni> jbrantly: when you pack the yabble.js script you can also handle its metadata!
[1:07] <jhuni> jbrantly: you don't need a server!
[1:10] <jbrantly> jhuni: you'd need to pack it. And that, for instance, is something you would ONLY do for browsers. And it adds requirements for tooling, which you may or may not want for quick development. Etc. I'm not saying its a crappy idea. I'm saying it's not for me. It goes against some of the goals of Yabble. And since it's browsers only, and optional, etc etc, I don't think it belongs in the CommonJS specification, that you have to, or that its r
[1:11] <jbrantly> jhuni: let me put it another way. Right now, it's possible for modules to be loaded into the browser via XHR and by <script /> tags. Some browser loaders use XHR. Some use script tags. Yabble happens to support both. But the point is, nowhere in the CommonJS spec does it say you have to or need to use one or the other. It's up to the implementation.
[1:12] <jbrantly> jhuni: so, back to your idea. It's not a bad idea. But it's up to the implementation.
[1:12] <Wes-> "you don't need a server" - you don't need a server with my monadic ensure strawman either, doesn't mean people will like it. :)
[1:13] <Wes-> The only part of require.ensure that needs a server, BTW, is the part where it magically knows it's dependencies
[1:13] <jbrantly> Wes-: I hate it :P
[1:13] <jbrantly> j/k
[1:13] <Wes-> jbrantly: That's neither here-nor-there, really -- the point is that "need a server" is only needed for automatic dependency resolution, I think
[1:14] <Wes-> Which is the real problem that needs solving, IMHO. Anything more is makeup and jewelery
[1:14] <jbrantly> Wes-: not sure how much of this you've been following, but thats one of the things we're trying to solve. Easier dependency discovery.
[1:15] <jbrantly> via "module metadata", which could include dependencies for example
[1:15] <Wes-> Haven't been following much, focus has been lousy, but thought jhuni was proposing general purpose meta data as a secondary file or something
[1:15] <jhuni> btw check out the Modules/Metadata page, I think it is more "lightweight"
[1:16] <Wes-> Listing dependencies manually is okay if you do it in a regular way, and on a module-by-module basis, one level deep
[1:16] <jbrantly> does anybody know if the implementation table takes a while to update on the wiki? Trying to figure out if I'm doing it wrong.
[1:16] <Wes-> conflating modules and packages in this regard gives me pause for concern
[1:17] <Wes-> You know what would work *really* easily in terms of dependency resolution, with minimal outside impact?
[1:17] <Wes-> exports.depends = [ "moduleA", "moduleB" ]
[1:18] <Wes-> bang! entire problem solved
[1:18] <jbrantly> Wes-: my idea, at least, is to have an optional way to specify metadata for a module, and one of the things that you could specify would be shallow dependencies. Yet to be decided are things like what exactly would go into the metadata, how it would be specified (your example just there, a separate file, etc), and whether or not its even needed at all
[1:18] <jhuni> Wes-: if you have a lot of dependencies it would be better not to include that with the file
[1:18] <Wes-> jhuni: why?
[1:19] <jhuni> Wes-: file size on the client-size
[1:19] <jhuni> s/client-size/client-side/
[1:19] <Wes-> jhuni: sorry, that's a better definition for better
[1:19] <Wes-> You have to transfer the same amount of information
[1:19] <Wes-> PLUS when you have two web hits, you have the overhead of two connections
[1:19] <Wes-> Plus, I challenge you to find a reasonable example of a module with say, more than 25 or 30 shallow depends
[1:20] <Wes-> s/a better/a poor/
[1:20] <jhuni> Wes-: not yet
[1:20] <Wes-> jhuni: Probably not ever, IMNSHO
[1:20] <jhuni> Wes-: btw I don't understand exactly what you mean about two web hits?
[1:20] <Wes-> jhuni: Just like the 2GB filesystem size limit isn't really a problem for C source code
[1:20] <Wes-> jhuni: transferring two files requires two HTTP requests
[1:20] <jbrantly> he means if it was in a separate file
[1:21] <Wes-> "if you have a lot of dependencies it would be better not to include that with the file"
[1:21] <jhuni> why would someone be having two http requests ever???
[1:21] <Wes-> So he means just not include it all?
[1:21] <jhuni> oh I mean not to include the dependencies data
[1:21] <jbrantly> Wes-: jhuni is not advocating a separate file. And yes.
[1:21] <Wes-> So, how are dependencies to be resolved?
[1:21] <jhuni> of course you should include the dependencies, but you don't need to include the data
[1:21] <jbrantly> no clue
[1:21] <Wes-> I still have problems with any answer that resembles hand waving or magic
[1:21] <jhuni> Wes-: on the server-side
[1:22] <Wes-> jhuni: Okay, that's the magical "by the transport" solutino
[1:22] <Wes-> Personally, I'm not fond of it, but I guess that's life
[1:22] <jbrantly> my problem is that everybody is still talking on totally separate playing fields
[1:22] <Wes-> (I think it should be possible to serve commonjs up from a minimalistic web server)
[1:23] <Wes-> jbrantly: because everybody who wants to solve the dependency problem thinks that they can use it to shoehorn something else in
[1:23] <Wes-> jbrantly: it's pork barrel specification writing!
[1:23] <jbrantly> Wes-: I completed my browser loader. No magic. http://github.com/jbrantly/yabble
[1:23] <jbrantly> Wes-: YES!!!!!!!!!!!! OMG YES!
[1:23] <jhuni> Wes-: if you want you could use the metadata to create a ensure transport, but as I said before that isn't the ideal solution for file-size/performance
[1:23] <Wes-> jbrantly: how do you resolve shallow deps?
[1:24] <jbrantly> Wes-: static analysis. Some failures of course, which is why I'm advocating a metadata specification.
[1:24] <jhuni> jbrantly: you like metadata?
[1:24] <jbrantly> Wes-: btw, "pork barrel specification writing" awesome
[1:24] <jhuni> jbrantly: but you wouldn't use it on your loader?
[1:24] <Wes-> jhuni: In the monadic ensure solution (which I am not advocating, btw) /all/ of the dependency information is naturally expressed and enforced by the source code, no meta data required
[1:24] <jbrantly> jhuni: yes, just not your solution.
[1:24] <Wes-> jbrantly: thanks. ;)
[1:25] <Wes-> jbrantly: static analysis -- not a huge fan, but it's not a terrible solution, either. CommonJS considered this when we were suggesting mandating only string literal arguments to require
[1:25] <jbrantly> Wes-: I just heard about the string literal argument. That of course is one point of failure. I didn't realize it had been considered already.
[1:26] <Wes-> Yeah. It's not SO bad, but it means you can't calculate module names effectively, which might be required in something like a versioning scheme.
[1:27] <Wes-> Making developers of modules responsible for lists of dependencies has the exact same problem, though
[1:27] <Wes-> because, if you can list it, the string literal solution can be made to work
[1:27] <jbrantly> Wes-: my thought about the metadata is that is very loose (read optional).
[1:27] <jbrantly> Wes-: there's actually another problem besides string literals
[1:28] <jbrantly> Wes-: going back one thought, the metadata is very optional. Meaning it might only be used when absolutely necessary.
[1:28] <Wes-> jbrantly: I had the same thought, actually, but it still doesn't solve all cases, unfortunately -- they all boil down to the string literal problem, I think
[1:28] <Wes-> Heck, I think possibly even Transports/C has that problem if you don't use the monadic ensure style
[1:28] <Wes-> of course, Transports/C *allows* the monadic ensure style
[1:28] <Wes-> It just doesn't mandate it, so you lose some of the guarantees
[1:28] <jbrantly> Wes-: forward one thought. The other problem is require.ensure. Consider this module: require.ensure(['moduleA'], function(require) { require('moduleA'); });
[1:29] <Wes-> Note for the picky, monadic ensure is not a monad
[1:29] <jbrantly> static analysis would pull out the require('moduleA'), but really its not a shallow dependency
[1:29] <jbrantly> since it's wrapped in the ensure
[1:30] <Wes-> That's not such a big deal, I don't think, so long as memoization happens as the start of the ensure
[1:30] <Wes-> which is the basic paradigm used on the server now, there is a test suite case which helps make sure it's right
[1:30] <jbrantly> well I mean. It's not a big deal in the sense that it won't cause an error. But it sucks if you're *trying* to get lazy-loading
[1:31] <jbrantly> I mean. I tend to agree. The problems are small and can be worked around. But I thought that a metadata proposal could be useful for the edge cases.
[1:31] <Wes-> So, you're suggesting that static analysis would be smart enough to pick out the require('moduleA') but not smart enough to figure out that it's in an ensure block?
[1:32] <jbrantly> Wes-: more or less. I use a heuristic now for dealing with it, but it's not smart enough to figure out that the require() is *inside* the ensure block. Just simple regexes.
[1:32] <Wes-> Yeah - to get this right, you need something which can generate a javascript parse tree
[1:33] <jbrantly> Wes-: right now I gather are modules that are require()d, all modules that are ensure()d, and subtract those in the second set from the first
[1:33] <jbrantly> Wes-: exactly :(
[1:33] <jbrantly> Wes-: which might be possible on the server. Not so much for a small client-side loader.
[1:33] <Wes-> here's the thing: why would a module mix ensure and non-ensured-requires?
[1:34] <jbrantly> Wes-: thats exactly my thought. Which is why I think it'll work in 99% of cases. But still...
[1:34] <Wes-> jbrantly: big tip for a client side loader: make sure you are generating regular, predicatable, HTTP GET URLs for your pulls so they can be cached! :)
[1:34] <jbrantly> Wes-: no crap :P
[1:35] <Wes-> jbrantly: If I was in your shoes, I would stop static analysis as soon as I saw an ensure statement
[1:35] <Wes-> As for caching, my thought is that you are probably loading more than one module per HTTP request?
[1:36] <jbrantly> Wes-: it's possible with server support
[1:36] <Wes-> Oh, crap, right
[1:36] <Wes-> *hmm*
[1:36] <jbrantly> Wes-: or with pre-compiled scripts
[1:36] <dmachi> I've written a couple of the serverside portion of this for dojo too, it works pretty well. The challenging part of keeping the requests cached was that there was a lot of data that the server either has to track (to know what you've loaded) or you have to send to the server
[1:36] <dmachi> but in the end it wasn't too hard and works pretty well.
[1:37] <Wes-> dmachi: do you need to send more than module names, in sorted order?
[1:37] <jbrantly> really, my thought is that for production, you probably will have a precompiled script
[1:37] <Wes-> jbrantly: and for dev, you can generate the precompiled script dynamically pretty easily
[1:37] <dmachi> Wes-: no, but that list can get big and not fit in 2k url for IE
[1:37] <jbrantly> Wes-: Yabble also supports just file-by-file loading for quick dev. Don't need to wrap modules or anything.
[1:38] <Wes-> dmachi: if (ie) alert('serves you right")
[1:38] <Wes-> dmachi: Actually, if you're doing it dynamically, you probably want to use relatively small lists to improve cacheability
[1:38] <dmachi> jbrantly: yeah though i basically just cache them so its bascially static after the first load. But i also had some optimizing routines that changed over time.
[1:38] <Wes-> I bet there's a sweet spot around 8-10 modules for typical installs
[1:39] <Wes-> The clever part would be getting good locality of reference to maximize cache hits
[1:39] <Wes-> You know
[1:39] <dmachi> 2-3 js requests it the sweetspot in the end, but the problem is that it varies over time, so i have a system that does exactly that
[1:39] <Wes-> putting many modules together probably would suck, now that I think about it, unless oyu have a totally optimized setup where you pretty much do 'em all
[1:40] <dmachi> in other words on the first page you go to the best request you can make is to get everyting in 1-2 requests at a max, and then when you go to the next page the "best" scenario is to load those 2 requests (cached) + whatever else you need
[1:40] <Wes-> tricky, though, imagine a page that loads 10 modules, and another that loads 5
[1:40] <Wes-> now, they both load env.js, which is 10,000 LOC. All the other modules are 25 lines of code each.
[1:40] <dmachi> So i have a degrading cache kind of thing that basically is optimizing your "layers" over time.
[1:40] <dmachi> Wes-: right, thats why i have to send to the server what the client already has
[1:40] <jbrantly> Wes-: thats really something that I've always had to more or less do by hand
[1:40] <Wes-> In that situtation, there is no way the client side could make a good aggregating choice to prevent env.js from getting loaded from the server twice
[1:41] <Wes-> dmachi: Ah! Yes!
[1:41] <dmachi> thats exactly what dojo's build system does for you, though i basiclaly wrote a version of that that works on the fly
[1:41] <Wes-> dmachi: Are you familiar with NNTP protocol? You've re-invented IHAVE SENDME
[1:41] <dmachi> i've actually written about 4 versions of it if i want to be honest
[1:41] <Wes-> (in a completely different context)
[1:41] <dmachi> hehe, yah. Not wanting to reinvent, just spent years on this stuff :)
[1:41] <Wes-> dmachi: what's your server-side written in?
[1:41] <jbrantly> dmachi: have you found that that works well? (sending known modules back to the server)
[1:42] <dmachi> i've written them in php, python (the good one was python), and partial js, which i'll redo once iget the current commonjs for titanium working
[1:42] <dmachi> jbrantly: it was pretty reliable for me yah, but you do have to deal with the issue of too many things somehow
[1:43] <Wes-> Nice. I'm trying to develop ALL new code here in JS
[1:43] <Wes-> When it works, it works amazingly well
[1:43] <dmachi> one of my customers basically used some sort of compression on th ename
[1:43] <Wes-> But I find myself spending a lot of time developing JS, instead of IN JS
[1:43] <dmachi> Wes-: me too, i'm 100% js on my personal dev
[1:43] <jbrantly> I think I need to start writing some plugins for Yabble, mainly around URLs for module loading so that you could load from static files or a script serving application (which may have its own URL API)
[1:43] <dmachi> Wes-: yeah :(
[1:43] <jbrantly> anyways, gotta go
[1:43] <jbrantly> later
[1:43] <Wes-> jbrantly: Good chat, see ya!
[1:43] <dmachi> jbrantly: ping me later an i can send you some of that stuff if you have any interest
[1:43] <Wes-> dmachi: FWIW, I found that a foreign function interface speeds things up nicely
[1:44] <Wes-> dmachi: Also, I found that being able to read PHP session files from JS helps me transition pieces
[1:44] <Wes-> (but web isn't really my focus, anyhow)
[1:44] <dmachi> My titanium port is going pretty well
[1:44] <dmachi> I almost have pintura running
[1:45] <dmachi> i'm using it for my test cases.
[1:45] <Wes-> titanium the perl thing?
[1:45] <dmachi> Will be pretty cool to be able to whip out desktop applications in no time
[1:45] <dmachi> appcelerator
[1:45] <Wes-> Hmm, don't know anything about that, unfortunately
[1:46] <Wes-> I'm still at a level where for my few web apps, I can get away with JS CGIs, just like perl in 1997 ;D
[1:46] <dmachi> its just basically webkit/jscore and you can build it for for mac, win, linux, iphone, android, etc
[1:46] <Wes-> Oh, okay, I'm vaguely familiar withi that
[1:46] <dmachi> what is your main goal/purpose from your code?
[1:47] <dmachi> I've just written a commonjs plugin for titanium so i can use all that code and get at the pintura data store easily to manage my data http://github.com/dmachi/Titanium-CommonJS
[1:47] <Wes-> dmachi: we use it for internal dev for a LOT of tasks -- basically, any place where non-speed-critical C code or shell scripts would go
[1:47] <Wes-> We also use it for a runtime configuration language
[1:47] <dmachi> i see, cool
[1:48] <Wes-> So, I guess I represent the Systems Programming crowd in a kind of funny way. :)
[1:48] <Wes-> I have been quite pleased with the performance of JS thus far (running spidermonkey) -- the expressiveness, of course, totally rocks
[1:49] <dmachi> nothing wrong with that :) I did systems work for a high performance computing center for a long time before i started this stuff
[1:49] <Wes-> I can actually dig down into C APIs deep enough that was i able to write a reactor with an event system that listens to TCP/IP sockets -- all in JS -- in a very short period of time. Sure beats bind/listen/accept/fork model
[1:49] <dmachi> Right now i'm actually writing a mini grid with titanium / common js (workers on vms or other machines and a queue) for testing apps
[1:49] <Wes-> dmachi: Nice!
[1:50] <Wes-> And last week I was doing line-level protocol stuff in JS
[1:50] <dmachi> I have a python version that isn't very deployment friendly thats for performance testing hundreds of urls, so this will make it easier
[1:50] <dmachi> kind of like personal selenium-grid :)
[1:51] <Wes-> That sounds awefully interesting
[1:52] <Wes-> "deployment friendly" - I can't wait to get there with GPSEE
[1:52] <dmachi> my whole 'port' has been pretty much all everyone elses code from narwhal/node/pintura
[1:52] <Wes-> I bumepd into a entertaining road block this week - we precompile our javascript down to byte code to get faster exec()
[1:52] <dmachi> sicnce they already have the low level written i just have to make shims :)
[1:52] <Wes-> The cache caches the inode number of the original script, for validation
[1:53] <Wes-> Talk about deployment unfriendly. :D
[1:53] <dmachi> hehe
[1:53] <dmachi> how do you do that?
[1:53] <Wes-> do what? The cache or the precompile?
[1:53] <dmachi> is it compile to bytecode and that bytecode is executed by gpsee or do you mean you compile the js to an executable?
[1:54] <dmachi> (i hadnt' heard of any actual complete version of the latter and seems a bit challenging when dealing wth some of the dynamic nature of js)
[1:54] <Wes-> That's right - we compile down to bytecode, which is the basic work unit of the spidermonkey engine. It will then either interpret the byte code, or turn it into assembler with the JIT
[1:54] <Wes-> The latter may actually be possible with v8 with a bit of work, but I don't work on V8 since it doesn't support sparc
[1:55] <Wes-> I used to actually fake executables a few years ago when I was running the NJS interpreter
[1:55] <dmachi> cool
[1:55] <Wes-> I had a build system that buried the byte code right in a dynamically generated executable which also contained an interpreter :)
[1:55] <dmachi> what sparc box are you still running?
[1:55] <dmachi> :D
[1:55] <Wes-> dmachi: our whole infrastructure is sparc except for the network layer
[1:56] <Wes-> Mostly worker bees like sunfire v240s - nothing fancy, just robust
[1:56] <dmachi> I had a sun e15k at that last job of mine, hardly got used
[1:56] <Wes-> *sigh*
[1:56] <Wes-> Those start at a million bucks
[1:56] <dmachi> the researchers at the time were all writing cluster type parallelizations instead of mpi and they couldn't make their apps scale more cost effectively than the clusters (not that it coudlnt' be done)
[1:57] <dmachi> yeah, ours was (at .edu discoutn) like 1.6...72 procs 288gb ram
[1:57] <Wes-> Interesting. All our work loads are basic transactions (not like OLTP, but analogous) - we don't need cross connects of any sort, really - so I just run lots and lots of boxes in parallel and switch between them based on chance or failure
[1:58] <dmachi> we had a couple domains though for oracle that it did really well with, but man i wouldn't wish oracle dba managment on anyone.
[1:58] * Wes- drools
[1:58] <Wes-> Of course, the big problem with e15k series outside of academia -- "single point of failure"
[1:58] <Wes-> I have to be online 24/7/365 without exception -- even a few seconds of downtime is a problem
[1:58] <dmachi> yeah, though it has a lot of built in redundancy
[1:58] <Wes-> So we'd have to have two if we had one. :)
[1:59] <dmachi> we did some hot switching/failover for domains
[1:59] <dmachi> but its not as big of a deal in research either
[1:59] <Wes-> dmachi: Yaeh, the built-in stuff is great, I just can't afford to hang my hat on it -- since it's a big "it"
[1:59] <dmachi> yep, agree
[2:00] <dmachi> do you use a shared fs?
[2:00] <Wes-> Yeah - in research, people get mad when stuff breaks. In my world, it's, ah much much worse. (Clue - I produce products for telecoms)
[2:00] <Wes-> dmachi: sun cluster global filesystem
[2:00] <Wes-> dmachi: pxfs, I think? analagous to nfs, but fewer hazards and totally transparent
[2:00] <dmachi> yeah, i can't remember the name of it, but we used one of those cluster fs as well
[2:01] <Wes-> neat thing about it - underlying storage can migrate from box to box (we have redundant fiber) without the applications ever noticing!
[2:01] <dmachi> we used sun grid engine too which was nice for submitting jobs across the cluster and the various smp boxes (and is awful useful in my current mini grid work :P)
[2:01] <Wes-> Yeah - read about that, neat stuff - not appropriate for my workloads, though
[2:02] <dmachi> nah, i mean it can do big coordinated things, it sbasically a queue, but more appropriate for things like this
[2:02] <Wes-> (I spend a lot of time trying to make my workloads independent... kind of like... immutable strings in a twisted brain path)
[2:03] <Wes-> submitting jobs across the cluster, these days, I have started to send JSON with wget to CGis written in gpsee
[2:03] <dmachi> yeah, isn't feasible on the research side. I mean we want to do that to the extent possible, but most of the problem sets require the computation and more importantly the memory.
[2:03] <Wes-> handy as all heck, I could even send code if I wanted to
[2:03] <Wes-> Yep. We're totally at the other end of the equation
[2:03] <Wes-> Just goes to show that it might be tough being a systems VAR :)
[2:03] <dmachi> Wes-: thats what i want to get too, though i dont' care to make a general job schedule and deal with all the risks, i want a limited set of jobs.
[2:04] <dmachi> hehe
[2:09] <tlrobinson_> what's the current winning transport proposal?
[2:22] <Wes-> tlrobinson: Transport/C, but I'm not sure you can call it winning, just "most recently discussed to death"
[2:42] <jbrantly> is it? For some reason I thought Transport/D was closer to being agreed upon
[2:52] <jhuni> I think I prefer Transport/A or Transport/C
[2:52] <jhuni> probably Transport/C
[3:05] <jhuni> jbrantly: btw I don't know why you think this metadata system isn't optional, it is optional, the only problem is determining what version of [[Modules/1.x]] the module is using, which is why I have a backwards-compatibility thing, so as to allow you to use an old version of [[Modules/]] such as [[Modules/1.0]]
[3:07] <jhuni> jbrantly: My current solution: if the module has metadata then it will be handled according to that version of [[Modules]], if it doesn't then assume its using [[Modules/1.1.1]]
[3:10] <kriszyp> yeah, transport/D is the only one that we have actually voted on
[3:10] <kriszyp> transport/C is more just a proposal based on what requireJS was doing
[3:11] <kriszyp> but transport/D is optimized for CommonJS module wrapping, and transport/C is optimized for hand-coded modules (that don't wrap an existing commonjs module)
[3:13] <jhuni> kriszyp: yet there is the issue with this transport system that a confused user might use something other then a string literal in a require() function
[3:14] <kriszyp> sure, but that is orthogonal to the transport specification
[3:14] <kriszyp> it is of interest to the implementors that are reading that the modules to transport
[3:14] <kriszyp> but doesn't affect the format they use to wrap the modules
[3:18] <jhuni> kriszyp: I am just saying I wouldn't recommend require to a beginning javascript user or to myself =/
[3:19] <kriszyp> you mean without telling them to use string literals?
[3:19] <jhuni> kriszyp: I mean I wouldn't recommend it at all, it is by nature synchronous
[3:20] <kriszyp> well I certainly would want to ever have to use an async require on the server
[3:20] <kriszyp> would not want...
[3:21] <jhuni> kriszyp: non-blocking IO has its benefits on the server too
[3:21] <kriszyp> not for program startup
[3:23] <jhuni> kriszyp: thats limiting yourself to loading things at startup
[3:23] <jhuni> some apps might like to load things at run-time
[3:24] <kriszyp> i have yet to see that. penalizing all other code for such a rare use case is silly
[3:24] <kriszyp> now, async on the browser is absolutely critical
[3:24] <kriszyp> I certainly don't deny that
[3:24] <kriszyp> so I certainly sympathize with the need for async module loading
[3:25] <jhuni> is there any reason you wouldn't want to use async loading on the server?
[3:25] <kriszyp> because it is extra typing, you have to write a callback
[3:25] <kriszyp> no thank you
[3:25] <jhuni> kriszyp: or a source filter
[3:25] <kriszyp> a source filter?
[3:26] <jhuni> kriszyp: things are being wrapped in a function anyways to give them lexical scoping
[3:26] <kriszyp> you mean that auto wraps it? with a callback?
[3:26] <kriszyp> sure, that is fine
[3:26] <jhuni> kriszyp: that is what I would like to use, a source filter
[3:26] <kriszyp> my web-modules project actually does async loading of commonjs modules kind of like that
[3:26] <kriszyp> but I don't have to code my modules with the callback
[3:27] <jhuni> kriszyp: have you seen this? http://wiki.commonjs.org/wiki/Modules/Metadata
[3:27] <jhuni> kriszyp: that is the system I would like to use for myself at least
[3:28] <jhuni> the part about "handlers" is about source filters, handlers is just a different name for source filter in this case
[3:28] <kriszyp> can you use a commonjs module in that loader?
[3:28] <kriszyp> it doesn't look like it is compatible with the commonjs module format
[3:28] <jhuni> kriszyp: if the module lacks metadata then it is assumed it is in commonjs module format
[3:29] <jhuni> kriszyp: that means the metadata is entirely optional, its a benefit to users and to servers that want to do optimizations
[3:39] <jhuni> kriszyp: do you like the idea of using JSON metadata to describe modules?
[3:39] <jhuni> kriszyp: I would like to build a metadata handler which wraps a script in Transport/D
[3:40] <jhuni> kriszyp: that way scripts can get metadata from require statements and then use metadata handlers to do the wrapping
[3:40] <kriszyp> I don't think it is necessarily a bad idea, i am just trying to figure out how it would jive with existing commonjs modules
[3:41] <jhuni> kriszyp: it would be a benefit to give data about those modules to anyone who can handle the data, also it would be nice to have a standard way of creating transports
[3:43] <kriszyp> how is it better than writing modules directly in transport/C or D?
[3:44] <jhuni> kriszyp: I thought you didn't like function wrappers =/
[3:44] <jhuni> kriszyp: JSON has its benefits, like JSON schema and JSON Query
[3:44] <kriszyp> I see, so it is better because it avoids the function wraper
[3:44] <jhuni> also the benefits of JSON
[3:45] <jhuni> you can choose between various transports
[3:45] <jhuni> without the string literals problem
[3:46] <jhuni> also you can do everything the current spec lets you do
[3:46] <kriszyp> I think it could be useful if it can be used as a progressive enhancement to modules
[3:47] <kriszyp> if I could go without it, but if for a particular module I want to give explicit up-front dep loading info, I can
[3:47] <kriszyp> but I would also want it to work with currently loaders, and I don't think that proposal is backwards compatible is it?
[3:47] <jhuni> kriszyp: what do you mean by backwards compatible?
[3:48] <jhuni> kriszyp: it doesn't work with current loaders, but you can make it work with require
[3:48] <kriszyp> there is no top-level META variable in current commonjs platforms
[3:48] <jhuni> I hadn't thought of that
[3:49] <kriszyp> although I guess one could wrap it in a check
[3:49] <kriszyp> if(typeof META !=="undefined"){META({...});}
[3:50] <jhuni> my implementation of META only needs asynchronous slurping to work
[3:51] <jhuni> it could be made to work on commonjs platforms if they have some asynchronous slurping api
[3:51] <jhuni> of course it could just as well be synchronous slurping
[3:53] <jhuni> http://github.com/jhuni/Module-Metadata/blob/master/lib/META.js
[4:02] * jhuni is going to think about how this system relates to other implementations/loaders
[4:43] * jhuni definitely thinks Modules/Transport/C is the best of the bunch
[14:05] <jbrantly> is there any reason why we can't add implementations to specs like Modules/1.1.1 or Async/A or have people just not gotten around to it yet?
[14:06] <dmachi> I think some of the specs do have implementations listed
[14:06] <dmachi> (though i'm not sure how current it is)
[14:07] <jbrantly> yea, I know, but certain ones don't even though I know they have implementations (again, Modules/1.1.1 and Async/A). My guess is that they just havent been updated yet, but wanted to see if I would be breaking process if I added some.
[14:08] <dmachi> I wouldn't think so, it is a wiki, but i dont' know for sure :)
[14:08] <jbrantly> two like minds are better than one. Good enough for me. :)
[14:09] <dmachi> based on our 1 agreed up on thing being debated yesterday, this might be some of the most conscensus there is :)
[14:10] <jbrantly> haha
[14:28] <ashb> jbrantly: you can add them, but unless its agrred upon widley it shold just be a 'trial implementation' i think
[14:29] <jbrantly> ashb: I figured that would be more or less assumed. For instance, if the spec itself is still in a "proposal" state, or there are no unit tests.
[14:30] <ashb> ask Dantman (when hes about) what the right template to use is
[14:30] <ashb> that or dig about in the wiki and try to find the rightonwe
[14:30] <jbrantly> I think I figured it out
[14:30] <jbrantly> wiki has been updated already
[14:30] <jbrantly> but I'll notify him so he can double-check me
[20:28] <Wes__> ash: ping
[20:28] <Wes__> ashb: ping
[20:30] <ashb> yo?
[20:30] <ashb> Wes__: ?
[20:31] <Wes__> ashb: What does perl do during the build process to get installed in /usr/bin/perl?
[20:31] <Wes__> ashb: Does it just go ahead and put a link to itself there?
[20:31] <ashb> you know i dont actually know...
[20:31] <ashb> i usually dont want it to do that
[20:31] <Wes__> HA, yeah
[20:32] <Wes__> I am trying to figure out how the commonjs distros can, by default, get commonjs binary into users' paths
[20:32] <Wes__> ln -s /usr/bin/commonjs /path/to/my/engine
[20:32] <Wes__> seems like something the build system could do relatively safely
[20:32] <Wes__> er
[20:32] <Wes__> ln -s /path/to/my/engine /usr/bin/commonjs
[20:33] <Wes__> maybe with a gentleman's agreement not to overwrite the link if it's already there
[20:34] <Wes__> ashb: what does flusspferd do?
[20:34] <ashb> /usr/local/bin/
[20:36] <Wes__> So it jams a commonjs program there, or flusspferd?
[20:36] <ashb> flusspferd
[20:37] <ashb> no `commonjs` itself
[20:37] <konobi> Wes__: it doesn't do it default
[20:37] <ashb> s/itself//
[20:37] <konobi> *by
[20:38] <konobi> there's a Configure option to have it install to /usr/bin/perl though
[20:38] <Wes__> konobi: thanks
[20:38] <konobi> but it'll always install under the supplied PREFIX
[20:38] <konobi> Wes__: checkout the INSTALL perldoc in the distro
[20:38] <Wes__> ashb: I think we should both agree to install $(PREFIX)/commonjs as a link to our respective engines, if there is no other link present
[20:38] <Wes__> er, $(PREFIX)/bin
[20:39] <Wes__> er, $(PREFIX)/bin/commonjs
[20:39] <Wes__> you know what I mean :)
[20:39] <Wes__> konobi: thanks for the pointer
[20:39] <hannesw> Wes__ on debian based systems there's the alternatives system
[20:39] <hannesw> http://www.debian-administration.org/articles/91
[20:39] <hannesw> lets you choose which packet/impl to use for a certain task
[20:40] <Wes__> hannesw: Hmm! That is definitely the way to get into /usr/bin on a Debian system, isn't it?
[20:40] <hannesw> yes
[20:40] <hannesw> but if we have a bin/commonjs, we need standard options too
[20:40] <ashb> hannesw: not quite :)
[20:40] <Wes__> hannesw: Standard options are already (sort of) taken care of: if we use /usr/bin/env, there can be none
[20:41] <ashb> http://code.google.com/p/gpsee/issues/detail?id=12
[20:41] <Wes__> hannesw: OTOH, v8cgi, flusspferd and GPSEE support comment-embedded options
[20:41] <Wes__> as ash points out :)
[20:41] <konobi> shebang is always good
[20:41] <hannesw> interesting, hadn't known about these
[20:41] <hannesw> well people might want to use commonjs explicitly
[20:42] <ashb> konobi: #!/usr/bin/env commons js -Oh-no-i-want-options
[20:43] <konobi> ashb: yeah... the options after shebang is a bit of a cross-platform minefield
[20:43] <ashb> aka don't work at all on linux
[20:43] <Wes__> or solaris
[20:44] <hannesw> doesn't work on debian/ubuntu, which sucks
[20:44] <ashb> i think it only works on BSDs, and its actually not posix compliant
[20:44] <Wes__> Hence the reason for comment-embedded options -- easy to parse before executing script, compatible with /usr/bin/env
[20:45] <ashb> and punts on the need for command line flags to be agreed upon
[20:45] <hannesw> i'll have a look at those, too
[20:45] <Wes__> Yeah, that's a bonus too
[20:45] <Wes__> "What colour is the shed?" "YOUR colour!" :)
[20:48] <konobi> well... #!/some/prefix/bin/commonjs --foo --bar --har
[20:48] <konobi> that works, iirc
[20:51] <Wes__> konobi: That works, except "what is /some/prefix" ?
[20:51] <Wes__> konobi: Ideally, it is either fixed on every system (/bin or /usr/bin) or in your path, hence /usr/bin/env
[20:52] <konobi> Wes__: I generally do the substition upon install
[20:52] <Wes__> konobi: You edit every program you download?
[20:53] <Wes__> bugzilla has that problem with perl
[20:53] <Wes__> you have to edit every single script (hundreds of them) if you don't want to use the perl in /usr/bin
[20:53] <konobi> Wes__: it's part of the make install targets for the software I distribute
[20:53] <mikeal> bugzilla needs to die
[20:53] <mikeal> forever
[20:53] <konobi> perl installer lets you specify which shebang line you want to use
[20:54] <konobi> (for the scripts that it installs)
[20:55] <Wes__> konobi: That still means that every single script you download now needs to come with an install program that can edit it
[20:55] <Wes__> Do all perl programs come with installers?
[20:55] <Wes__> I guess CPAN can handle a lot of that
[20:55] <konobi> CPAN gives you the tools for that
[20:56] <konobi> `perl Makefile.PL; make; make test; make install`
[20:56] <Wes__> Maybe perl is a bad example here
[20:56] <Wes__> How about python and ruby?
[20:57] <konobi> python setuptools can give you something like that iirc, not sure about ruby
[21:00] * konobi is surprisingly tired after jsconf
[21:04] <Dantman> ruby you usually have a `sudo ruby setup.rb` or you use rubygems
[21:05] <Dantman> Actually RubyGems probably runs that as well.
[21:05] <Dantman> Damn you WebKit and your type=number lies!
[21:06] <Wes__> wow, I can't believe how hard other programming languages make running scripts
[21:06] <Wes__> I really, truly believe that this should work:
[21:07] <Wes__> wget http://scriptorium.com/scripts/fibonacci.js
[21:07] <Wes__> chmod 755 ./fibonacci.js
[21:07] <Wes__> ./fibonacci.js
[21:07] <Wes__> and this should ALSO work
[21:07] <Wes__> commonjs fibonacci.js
[21:08] <Dantman> Eh?
[21:09] <Dantman> There's nothing stopping you from `wget http://example.com/scripts/fibonacci.rb; ruby fibonacci.rb`
[21:10] <konobi> Wes__: you're really talking about a gap in the POSIX spec... not a language issue
[21:10] <Dantman> Most of the python and ruby scripts I see anyways don't include shebang lines
[21:11] <Dantman> And js scripts won't either for good reason...
[21:12] <Dantman> ((# isn't a valid comment in js, there are too many different interpreters, half the scripts are meant to work in a browser, and absolute paths fail))
[21:29] <kriskowal> Dantman we strip shebang as part of the loading process in Narwhal.
[21:29] <kriskowal> i presume everyone else is doing that too
[21:30] <kriskowal> hannesw_, Wes__, ashb i was thinking we need a "Scripting/A" proposal
[21:30] <kriskowal> of course, i've been thinking no small number of things since jsconf
[21:30] <kriskowal> so i can guarantee that i won't get around to writing such a thing in a hurry.
[21:30] <Dantman> kriskowal, My plan was the same... however that means that your script won't work in browsers, and it's perfectly reasonable for it to not work in other commonjs implementations, we never said shebang lines should work.
[21:31] <kriskowal> yes, i think that's fine.
[21:31] <kriskowal> you wouldn't put a shebang in a non-cli main module
[21:31] <hannesw> kriskowal what would "scripting/a" be?
[21:32] <kriskowal> a proposal for a commonjs bin that can be used in a shebang, and probably some standard args
[21:32] <kriskowal> options
[21:32] <hannesw> ah ok. yes
[21:32] <kriskowal> with some additional constraints for the file-system based module loader that would apply in that situation
[21:33] <hannesw> so jsconf was inspiring?
[21:33] <kriskowal> like canonicalizing the script path and having it on system.args[0] or system.command or something
[21:33] <kriskowal> it was a lot of things
[21:33] <hannesw> :)
[21:33] <kriskowal> but the short story is that my todo list is longer than before
[21:34] <kriskowal> commonjs module proliferation is good.
[21:35] <kriskowal> and most folks on the ground i talked to like what we've done. some are unhappy that we appear to be stalled on async issues.
[21:35] <kriskowal> so i've gotta write a roadmap
[21:36] <hannesw> yes, there's something like a deadlock around the sync/async thing
[21:36] <kriskowal> there's disagreement about how quickly we should be moving. some are frustrated we're not moving fast enough, some thing we're stalled and not delivering
[21:36] <kriskowal> some think our effort is too early
[21:37] <hannesw> i don't think so
[21:37] <kriskowal> yeah, i think we're on time
[21:38] <kriskowal> but we need to adjust our strategy in naming and "ratification" so that we don't paint ourselves into a corner
[21:39] <hannesw> can you explain some more?
[21:39] <kriskowal> i'll write a blog
[21:39] <hannesw> cool
[21:40] <kriskowal> but the short story?
[21:40] <deanlandolt> kriskowal: i remember some general agreement that we shouldn't "ratify" anything...multiple implementations /is/ ratification
[21:40] <kriskowal> we perhaps don't need to agree on as many things. the whole vote before we proceed thing might not be necessary
[21:40] <deanlandolt> okay, so yeah, that's what i remember
[21:40] <kriskowal> right, so making that work means that our proposals need unique names
[21:40] <deanlandolt> btw: it was great meeting you last weekend...a lot of fun!
[21:41] <kriskowal> yeah, awesome to meet you
[21:41] <kriskowal> like, "binary-b" and "binary-f" can coexist
[21:41] <kriskowal> but they'd have to have names like that.
[21:41] <kriskowal> or require("system-1.0")
[21:41] <hannesw> ok, get it
[21:42] <kriskowal> and stuff that hasn't been proposed needs to get off the lawn
[21:43] <deanlandolt> heh!
[21:43] <kriskowal> like, i moved "sandbox" to "narwhal/sandbox".
[21:43] <deanlandolt> kriskowal: any thoughts of how modules would eventually migrate to "binary"?
[21:43] <kriskowal> well, in a way it's too late to migrate away from "binary"
[21:44] <kriskowal> but, like ry suggested on the commonjs panel, it could happen when there's a clear consensus among implementations
[21:44] <kriskowal> just like border-radius
[21:44] <ashb> kriskowal: we strip shebang too, yes
[21:45] <deanlandolt> yeah...but in the meantime we're gonna have a whole lot of -moz- cruft :-/
[21:46] <deanlandolt> hmm...i wonder if notes in package.json could be used to peg you to a spec
[21:46] <kriskowal> i think we have that already
[21:46] <kriskowal> there's an "implements" property
[21:46] <deanlandolt> ah, cool
[21:47] <Dantman> border-radius is /fun/ 99% of the people using it don't support every browser that has it implemented.
[21:48] <hannesw> is there a video online for that jsconf commonjs panel?
[21:49] <kriskowal> there prolly will be eventually
[21:49] <kriskowal> it was quick.
[21:49] <deanlandolt> hannesw: usually takes them some time
[21:49] <kriskowal> my talk at jsconfeu never got posted
[21:49] <kriskowal> most talks, really
[21:49] <kriskowal> because of the amount of time and energy they put into post-production

 

 

Logs by date :