Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2009-11-17:

[0:40] <isaacs> anyone ever played with jscc?
[0:40] <isaacs> seems kinda like ragel/antlr, but for/in javascript
[0:40] <isaacs> http://jscc.jmksf.com/
[0:41] <ashb> isaacs: someone in here has. they hit a few bugs but were able to work around it
[0:42] <ashb> isaacs: tho the code style/docs/tests (or lack of?) didn't inspire me with confidence
[0:42] <isaacs> yeah, me neither
[0:42] <isaacs> but i'm shopping around for a better approach for yuicompressor.
[0:42] <ashb> i've been seriously tempted to start writing either a better L[LR] generator or a packrat parser
[0:43] <isaacs> antlr and ragel seem to be the top parsers on the market
[0:43] <ashb> in terms of?
[0:43] <isaacs> power and community.
[0:43] <isaacs> and stability
[0:43] <ashb> aht about flex/bison? ;)
[0:43] <isaacs> haven't looked at it, but it's on the list.
[0:43] <ashb> not heard of ragel before
[0:44] <isaacs> the only one i've got any experience with is lex/yacc, and 1) that was back in college and 2) it made me stab my eyes out.
[0:44] <isaacs> ragel is popular in ruby circles
[0:44] <isaacs> it outputs a very strict state machine in the target language (goto, switch, and not much else), which is kinda cool for speed.
[0:45] <ashb> ah
[0:45] <isaacs> but i'd like to get as much of the functionality as possible in a language that is friendly to look at and work with
[0:46] <isaacs> part of my beef with yuicompressor today is the heavy reliance on java
[0:46] <isaacs> a javascript tool should be in javascript, imo
[0:46] <isaacs> (as much as possible, anyhow)
[0:46] <ashb> yeah.
[0:46] <ashb> i do have plans (genreally) to write some form of parser genreator
[0:46] <isaacs> antlr is java, but can output to js as a target language.
[0:47] <ashb> probably wont be LALR to start with as they are harder to write
[0:47] <ashb> last time i used antlr was at colleg. Take haskell, turn it into C
[0:47] <ashb> that was 'interesting'
[0:50] <isaacs> if i end up using antlr, that won't be the end of hte owrld, of course.
[0:50] <isaacs> the code i end up supporting can still be mostly js.
[0:51] <ashb> yeah
[0:51] <isaacs> but it'd be cool if the outcome of this was some kind of commonjs thing
[0:51] <ashb> it would just feel cooler to use JS all the way?
[0:51] <isaacs> definitely!
[0:51] <ashb> so a parser genreator is probably more than one persons worht of work
[0:51] <isaacs> absolutely
[0:51] <ashb> i'm *certainly* interested in writing one
[0:51] <_ry> a good js parser generator would be great
[0:51] <ashb> cool. thats 3 of us :D
[0:52] <isaacs> ashb: i'm not interested in working with one that isnt' already written :P
[0:52] <ashb> isaacs: bah.
[0:52] <isaacs> hehe
[0:52] <ashb> where's your sense of adventure?
[0:52] <isaacs> ashb: make it antlr compatible?
[0:52] <ashb> do you havea preference for kind?
[0:52] <isaacs> not particularly
[0:52] <ashb> (rec-decent, packrat, LL, LR?
[0:52] <_ry> i've tried jscc, wasn't happy. i forget why
[0:53] <ashb> i didn't get as far as trying
[0:53] <isaacs> anything that can parse js and css and give me a good way to compress them in mostly correct ways
[0:53] <ashb> it just seemd... wrong
[0:53] <isaacs> _ry: i'm looking ati ti now. the grammar syntax seems odd.
[0:53] <ashb> that was part of it
[0:53] <ashb> i'm quite taken by something like treetop's syntax
[0:53] <_ry> ashb: yeah
[0:54] <isaacs> interesting
[0:54] <isaacs> ruby certainly < java, imo
[0:54] <isaacs> er... ruby > java
[0:54] <_ry> the most important thing about a parser, imho, is that i can be interruptable
[0:54] <_ry> which treetop parsers arn't
[0:54] <ashb> interruptable?
[0:54] <ashb> oh, 'i need more data' type of thing?
[0:55] <_ry> that drinking wine, forgive the spelling
[0:55] <_ry> that you can give it a partial string, and it can deal with that
[0:55] <_ry> then give it the rest later
[0:55] <_ry> like from a socket
[0:55] <isaacs> _ry: that would be key if i was to tie it into node.js
[0:55] <ashb> thats a hard problem. since the last X productions could need reparsing
[0:56] <ashb> or up to the last top level rule, i.e. a statment in JS or CSS
[0:56] <_ry> of course you can always buffer data and then parse it - but that wastes memory
[0:56] <_ry> better if you can just do it
[0:57] <_ry> some sort of treetop-like system would be rad though
[0:57] <ashb> and shift-reduce parsers can just 'stop' and store state when they get to the 'EoI but not EoF' token?
[0:57] <ashb> http://github.com/tobie/pdoc/blob/master/lib/pdoc/parser/treetop_files/ebnf_javascript.treetop
[0:57] <ashb> i <3 that syntax for some reason
[0:57] <isaacs> yeah
[0:57] <_ry> i think an http parser, for example, could be done in js/v8 that was almost as fast as the C version
[0:58] <ashb> nah - even a shift-reduce parser would need to reparse something
[0:58] <ashb> since look ahead cna change things
[0:58] <_ry> ashb: maybe it can be smart about what it buffers?
[0:59] <ashb> that kind of smat's ins't the job of the parser but of annotations in the grammar i think
[0:59] <ashb> unless.
[1:00] <ashb> once you get to tokenizing/amtching on EOf you just stop right hten and wait for more input.
[1:00] <ashb> tho i guess things like JS could still be ambigious
[1:00] <ashb> case:
[1:00] <ashb> "var a = 1" " + 2";
[1:00] <ashb> due to Js's semicolon rules that first chunk is 'valid'
[1:01] <ashb> so you have to annotate what are suitabke 'stopping' points i think
[1:02] <ashb> (i think?)
[1:02] <ashb> and if you can't match up to a stopping point, you throw aray the last productions up to that point, and when oyu get more data you repase the last token
[1:03] <ashb> (i've been thinking alot about this problem recently. not stoping, but parsing in JS)
[1:03] <kriskowal> isaacs i have a branch of brendan's narcissus on github
[1:04] <isaacs> kriskowal: narcissus is on my list
[1:04] <isaacs> here's what i need:
[1:04] <kriskowal> it has a complete lexer, parser, generates an AST
[1:04] <ashb> kriskowal: intersting
[1:04] <isaacs> right, i need that for CSS, too
[1:04] <kriskowal> also notes var hosting
[1:04] <kriskowal> *hoisting
[1:04] <kriskowal> but it's not 100%
[1:04] <ashb> what do you mean by that?
[1:05] <isaacs> i need to get the AST, and then be able to write functionality to intelligently munge it, and have it spit out that AST in code.
[1:05] <kriskowal> it doesn't have a rewriter, which would be the bulk of a minifier
[1:05] <isaacs> right
[1:05] <kriskowal> so, i think that's a good place to start.
[1:05] <isaacs> so, antlr can do that, and SAC implementations pretend do, but don't really get there.
[1:05] <isaacs> basically, you can hook into the parser, and write functions to handle the tokens as they come in, or something like that.
[1:06] <ashb> kriskowal: you removed the SM specifcss in narcisus and replcaed then with ES5isms?
[1:06] <kriskowal> i was thinking of rewriting the parser/lexer to use an iteration approach, but i ended up doing other things.
[1:06] <kriskowal> not all of them
[1:06] <kriskowal> i didn't even touch the executer, where most of that goes on
[1:06] <kriskowal> it would not be hard to shift to defineProperties
[2:13] <okito> hey kriskowal yt?
[2:13] <kriskowal> aye
[2:13] <okito> I'm thinking of making a change in Narwhal and I wanted to run it by you first
[2:13] <okito> basically I am trying to integrate the loader we use in SproutCore (in the web browser) with narwhal
[2:14] <kriskowal> ok
[2:14] <okito> so that any packages [frameworks in SproutCore lingo] written for SproutCore could conceivably load with narwhal once they are updated to use modules
[2:15] <okito> the issue is that my build tools actually provide some extra glue code around each file
[2:15] <okito> automatically handling imports and exports if you want them
[2:15] <okito> so my idea was to allow the package.json to specify a "loader"
[2:15] <kriskowal> are they .js files?
[2:15] <okito> yes.
[2:15] <kriskowal> hm
[2:15] <okito> but they have import and export statements at the top
[2:15] <okito> i.e.
[2:15] <kriskowal> we have some multiplexing code for other loaders, but they key on extension so far
[2:15] <okito> right
[2:15] <okito> I noticed
[2:16] <okito> so I was thinking of modifying this code so that maybe if you don't name a loader by extension (which seems to be designed for native code), then it will use the loader named in the package.json
[2:16] <okito> if specified
[2:17] <kriskowal> atm, the package system and the module loader are wholly decoupled
[2:17] <okito> is that a design goal or just how it happens to work?
[2:17] <kriskowal> no information about package origin passes into the loader
[2:18] <okito> I could still keep it decoupled actually
[2:18] <okito> the package system could basically register loaders against paths
[2:18] <kriskowal> it's not so much a design goal, as there are certain benefits to the arrangement
[2:18] <kriskowal> like, compatibility with other loaders
[2:18] <okito> so do you intend for a dev to be able to swap out loaders?
[2:19] <kriskowal> like, we also want to eventually support helma as an engine; the package manager would be compatible as is since it only modifies the require.paths list.
[2:19] <kriskowal> objj just adds itself to require.loader.loaders
[2:19] <okito> that is how it handles .j files
[2:19] <kriskowal> right
[2:19] <okito> but since these files are actually regular js
[2:19] <kriskowal> *but* on the other hand, there may be a time when we want to support module.package metadata
[2:20] <kriskowal> yeah, i follow
[2:20] * kriskowal thinks
[2:21] <okito> OK so what if I modified the package module to simply add a global somewhere (system maybe -- just like paths) mapping path -> loader
[2:21] <okito> if the loader respects it great
[2:21] <okito> if not, it just won't work
[2:22] <kriskowal> you could replace the .js loader with something that switches on that data
[2:22] <okito> well it would be nice for ppl to be able to publish sproutcore-based packages via tusk
[2:22] <okito> and load them in any narwhal install
[2:22] <kriskowal> yeah, i follow
[2:23] <kriskowal> so, you could make a sprout loader package for them to depend on?
[2:23] <okito> right. that was the idea.
[2:23] <kriskowal> we have support for "preload" in package.json
[2:23] <okito> which does what?
[2:23] <kriskowal> loads a module at boot time
[2:23] <kriskowal> objj uses it to register its loader
[2:24] <okito> hm i see
[2:24] <kriskowal> so you can use that it inject your loader, by walking require.loader.loaders and replacing the [".js", loader] with yours
[2:24] <okito> so then I would swap out the loader at that time
[2:24] <okito> would that still satisfy your requirement for Helma?
[2:24] <okito> cool idea
[2:24] <kriskowal> well, your modules wouldn't work in helma, but that's another story.
[2:24] <kriskowal> objj won't work in helma either; that's okay
[2:24] <okito> ok
[2:24] <okito> maybe I don't care
[2:24] <okito> :)
[2:25] <okito> sounds like that might get me to the next step
[2:25] <kriskowal> what kind of boilerplate are we talking about here?
[2:25] <okito> one other question, where should I put code inside of a package that should be used browser-only vs narwhal-only?
[2:25] <kriskowal> in existing sproutcore frameworks
[2:25] <okito> one sec...I'll pastie something...
[2:26] <kriskowal> oh, if you want something to be browser only, use the engines tree
[2:26] <kriskowal> engine/browser/lib
[2:26] <okito> so inside of a package I put engine/foo
[2:26] <okito> ok
[2:26] <okito> and I guess engine/default contains files that will be used if engine/foo does not override it?
[2:27] <kriskowal> i'm betting that your boilerplate in sprout is functionally equivalent to our module-trasport-format
[2:27] <kriskowal> so it would be cool if we could converge on that
[2:27] <kriskowal> so commonjs modules complied or hand-written for browsers in that format could be loaded server-side too, to complete the loop
[2:28] <kriskowal> the packages code constructs a require.paths that includes engines/{engine}/lib, engines/default/lib, and lib, for each package
[2:28] <kriskowal> in that order
[2:31] <okito> I agree.
[2:31] <okito> I've been working with dangoor to develop this btw.
[2:31] <okito> He is using it for Bespin plugins
[2:31] <okito> Here is an example:
[2:31] <okito> http://gist.github.com/236570
[2:32] <okito> the other thing that is not obvious from the code snippet is that the explicitly named imports/exports allows the build system to determine dependencies when packing JS to send to the browser
[2:34] <okito> re: paths - got it
[2:34] <kriskowal> second arg is ihab's package shortname?
[2:34] <okito> yes
[2:34] <okito> actually my current code also allows a derivative form
[2:34] <okito> you can do
[2:35] <okito> 'hello_world:core'
[2:35] <okito> to explicitly name a package
[2:35] <okito> this way you can do "import hello_world:core"
[2:35] <okito> but that is not as important. just convenience.
[2:36] <kriskowal> how are these compiled?
[2:36] <okito> the sproutcore build tools: http://github.com/sproutit/sproutcore-abbot
[2:36] <kriskowal> i mean, via regex scan?
[2:37] <kriskowal> for the initial string literals?
[2:37] <okito> oh well yes
[2:37] <okito> primitive regex + state machine
[2:37] <kriskowal> yeah
[2:37] <okito> the string literals must appear before all code
[2:37] <okito> once I encounter anything that is not a free-standing string or comment I stop parsing
[2:37] <okito> the idea was to build on the "use strict"; idea from ES5
[2:38] <okito> I also allow other compiler flags
[2:38] <kriskowal> yeah
[2:38] <okito> "use modules false"; for example turns off module wrapping
[2:38] <okito> which is mostly used for bootstrapping
[2:38] <kriskowal> for imports and exports, i'm largely of the opinion that this is a bit of a waste, but the only harm is that commonjs compatibility becomes a one-way street to sprout
[2:39] <okito> well you don't have to use them
[2:39] <okito> I mostly added them because we have 200,000+ lines of JS we will need to convert
[2:39] <okito> this is simpler than making people explicitly add glue code around it
[2:39] <kriskowal> and they're already in this format?
[2:39] <okito> mostly
[2:40] <okito> they are optional though
[2:40] <kriskowal> in any case, the compiled output could converge with narwhal's module transport format.
[2:40] <okito> that was exactly my thought
[2:40] <okito> hence the loader
[2:40] <okito> if you want to use this glue-code stuff, switch to this other loader
[2:40] <okito> the same loader could process the build directives and produce the function body
[2:40] <kriskowal> i mean, we compile our modules to look like require.register{id: {factory: function(r,m,e){?},depends:[]}}
[2:41] <okito> yeah my loader is similar but a little different
[2:41] <okito> I register packages
[2:41] <kriskowal> which is your module factory function + id + dependencies
[2:41] <okito> and then register modules within the packages
[2:41] <okito> I started following that model actually
[2:42] <kriskowal> having all that together makes it possible to do require.async with or without a preload list
[2:42] <okito> but i had to move away from it because combining multiple modules was complicated.
[2:42] <kriskowal> oh yeah? what problem did you run into?
[2:42] <okito> well my API like I said is similar but instead of bundling all the items into a single register call
[2:42] <kriskowal> i had presumed that concatenation or combination into the main hash would work
[2:43] <okito> yes. I didn't want to build the hash in memory and then serialize it
[2:43] <okito> so the API right now looks like this:
[2:43] <okito> tiki.register('hello_world', { // package description, with dependencies })
[2:43] <okito> then for each module:
[2:43] <okito> tiki.module('core', 'hello_world', function() { .... })
[2:44] <okito> also each individual script file you load ends with:
[2:44] <okito> tiki.script(scriptId)
[2:44] <okito> tiki is my loader's name btw. It could easily be require; I just didn't want to take that global name just yet
[2:44] <okito> anyway, I could use the other method also
[2:44] <okito> the code is in flux. :)
[2:45] <kriskowal> sure. i haven't thought enough about how packages complicate things; what have you discovered there?
[2:45] <okito> oh I was saying - by breaking the API into three calls instead of one it means that I have serve my package modules in a single file or multiple files
[2:45] <okito> well if you are loading a lot of them namespacing becomes a serious issue
[2:45] <okito> b/c its too easy for different devs to overwrite module names
[2:46] <kriskowal> that's a symptom of narwhal package style, yes.
[2:46] <kriskowal> it isn't a symptom of ihab's proposal tho, if you do it right
[2:46] <okito> b/c of the package name in the require()?
[2:46] <okito> that is why I implemented that.
[2:47] <kriskowal> the canonical identifier for a module becomes [packageId, moduleId]
[2:47] <okito> right
[2:47] <kriskowal> and the packageId is a URL
[2:47] <okito> in my case I simplified that to "packageId:moduleId"
[2:47] <kriskowal> aliased with a short-name in package.json
[2:47] <okito> I don't think the URL is the best thing though
[2:47] <okito> because often times the URL you host at might change
[2:48] <okito> for example, if I might load some common packages from a CDN
[2:48] <okito> on another host
[2:48] <okito> then load some for myself
[2:48] <okito> so I think its better to have a canonical ID that may or may not be the url
[2:48] <kriskowal> sure, but that only makes caching difficult, doesn't break them. you do have to recompile tho.
[2:48] <okito> well that makes it pretty fragile.
[2:48] <kriskowal> sure, but then you don't really solve the name-space issue, and you get collisions among developers; is that not what you're saying?
[2:49] <okito> the other thing is that sometimes I might want to actually load the same package twice
[2:49] <okito> because it happens to be carried in two JS files
[2:49] <okito> I need to be able to detect that a particular package is already loaded and ignore it
[2:49] <okito> well my packageId's can be namespaced
[2:49] <okito> er nested
[2:49] <kriskowal> twice from two different urls with the same canonical name?
[2:49] <okito> yes.
[2:50] <kriskowal> thats?wasteful. how would that happen?
[2:50] <okito> not necessarily
[2:50] <kriskowal> two developers not bon the same page?
[2:50] <kriskowal> *on
[2:50] <okito> for example, to get an app up and running as fast as possible I might create a custom JS file that has just the modules I need to get up and running
[2:50] <okito> then I load the full set of packages in the background.
[2:50] <okito> as a separate file
[2:50] <okito> to maximize caching, it would be best to have a single JS file for background loading shared across all my apps
[2:51] <okito> but then each of this quick-start JS files are totally different
[2:51] <okito> so even though I may load a small amount of code twice, it will actually be faster overall for the user
[2:54] <okito> dang - wife is calling. I have to go.
[2:54] <okito> thanks for the help tonight. we should chat more about this.
[2:54] <kriskowal> aye
[2:54] <okito> the browser loader I'm working on btw is completely separate from SproutCore:
[2:54] <okito> http://github.com/sproutit/tiki
[2:55] <okito> I'm trying to make this low-level work independent so its not all SC or nothing. :)
[2:56] <okito> ciao
[3:23] <inimino> ashb: I have a PEG parser generator in pure JavaScript, and an ES5 grammar for it
[3:24] <inimino> needs to be cleaned up a bit before I want to publish it, but it works
[3:54] <Dantman> Agh, brain pinwheel... Java and it's lack of unsigned doesn't make it easy for me to figure out how to convert between the binary representation of a signed and an unsigned number in js space
[4:43] <kriskowal> Dantman http://github.com/280north/narwhal/blob/master/engines/rhino/lib/binary-engine.js#L17-23
[4:50] <Dantman> Yay, now to convert that to a .unsign() and .sign() method set.
[4:51] <kriskowal> that return byte strings?
[4:52] <Dantman> No, this is just something in pure Rhino...
[4:52] <Dantman> I'm just adding 2 methods to the Number prototype to make handling unsigned easier
[4:55] <Dantman> Bleh... T_T I have to deal with a 32bit unsigned int
[4:56] <Dantman> Omitting unsigned from Java is one thing, but completely omitting a binary unpacking api? ugh,
[4:57] <Dantman> I can't believe no-one has even written one
[5:10] <Dantman> *sigh* There is one... but ugh
[5:11] <Dantman> The java.nio.ByteBuffer can be used for that... But creating a ByteBuffer to convert a few bytes you already have?
[5:13] <Dantman> ((But of course, that API is completely signed ^_^, we all /love/ java!))
[5:15] <Dantman> Bleh, BSON is little-endian... Java is big-endian... Well, at least ByteBuffer supports switching as an option, and I only need to bother with one 4byte int
[5:25] <Dantman> kriskowal, lemme guess... that formula is incorrect for working with an unsigned 32bit int, right?
[5:30] <kriskowal> those formulae are correct for translating between signed and unsigned 32 bit integers
[5:32] <Dantman> ^_^ yay, so I don't have to do anything extra to make .sign() and .unsign() work in both cases where I need it
[14:38] <kriszyp> are there any IDEs that have added support for understanding CommonJS modules (for code completion)?
[14:41] <kriszyp> or are there any editor integrations with rhino's debugger available?
[14:47] <ashb> not that i'm aware of
[14:48] <kriszyp> I am just looking at the code for rhino's visual debugger, looks tantalizingly easy to just add editing capabilities myself...
[14:48] * kriszyp doesn't usually find writing his own IDE to be productive :/
[14:48] <ashb> no, that seems... foolish :)
[14:49] <ashb> i should get these doc changes of mine integrated and finish mongodb
[14:49] <ashb> then i can work on something cooler
[14:49] <kriszyp> heh
[14:49] <ashb> like a debugger
[14:49] <kriszyp> I wonder if hannesw is around, he has probably looked into this...
[14:50] <ashb> or one of the 27 other issues we've still got open for the next release >_<
[14:50] <ashb> http://redmine.flusspferd.org/versions/show/8
[14:50] <kriszyp> if you write a debugger where you can edit the files in the debugger... that would be very compelling
[14:50] <ashb> i wouldn't ever do that - I've got vim
[14:50] <ashb> and also: its *hard*
[14:50] <ashb> you can't change the source thats being run mid compile with any of the JS engines
[14:51] <ashb> *mid execution
[14:51] <ashb> not without going very machine+engine specific
[14:51] <kriszyp> Being able to edit right from a breakpoint though, I don't care how bad the editor is, that would be a huge productivity win for me, I think
[14:51] <kriszyp> and sure, you ahve to execute antoher request to see the change, but that is fine with me
[14:57] <ashb> hrmmmm
[14:57] <ashb> first step is to get a working debugger ;)
[14:58] <ashb> none of those issues (in hippo 0.9) are hugely exciting from a user pov
[15:08] <MisterN> ashb: if you're not a c++ embedder.
[15:09] <MisterN> i think for c++ embedders they are quite exciting.
[15:09] <ashb> yeah, as a javascripter they aren't tho
[15:32] <MisterN> ashb: if they want to write c++ modules, they are :P
[15:33] <MisterN> unfortunately people like you say "omg i don't want to do this at all!"
[15:35] <kriszyp> woot, I have editing in rhino visual debugger working!
[15:44] <ashb> MisterN: i dont mind it and quite enjoy it sometimes, but its not my end purpose
[15:45] <MisterN> ashb: you do gain from it becoming easier.
[15:45] <ashb> sure do :D
[15:54] <ashb> ah so you just upload source archives to the PPA. thats cool
[16:00] <MisterN> ashb: the PPA compiles automatically?
[16:03] <ashb> MisterN: yeah it seems to
[16:04] <MisterN> ashb: that might simplify things.
[16:05] <MisterN> ashb: i definitely would love to see a flusspferd 0.9 ubuntu package
[16:05] <ashb> yeah
[16:05] <ashb> i thought i'd try with spidermonkey first
[16:05] <ashb> cos you know, that's "easier" right?
[16:06] <ashb> >_<
[16:06] <MisterN> ashb: no, cause fp has sm as a dep!
[16:08] <ashb> yeah, the PPA uses xen VMs to build things
[16:14] <MisterN> nifty
[20:05] <ashb> http://openwebfoundation.org/2009/11/introducing-the-open-web-foundation-agreement.html
[22:01] <ashb> http://bulknews.typepad.com/blog/2009/11/dynamically-enabling-plack-middleware.html
[23:50] <ashb> deanlandolt: http://bulknews.typepad.com/blog/2009/11/dynamically-enabling-plack-middleware.html
[23:50] <ashb> incase you missed my earlier post

 

 

Logs by date :