Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2010-02-26:

[9:28] <Dantman> T_T Stupid web browsers... I started to have some fun thinking about allowing files to be uploaded in partially supported formats like jpeg2000, apng, maybe even TIFF. Checking the Accepts header for browser support, and without support instead sending the image in a format that the browser can handle... ie: Downgrading jpeg2000 to jpeg, apng to gif, and so on...
[9:28] <Dantman> But apparently no browser sends good competent accepts headers that way...
[9:32] <ondras> Wes-: ?
[9:32] <ondras> ashb: ?
[9:44] <Dantman> ^_^ Looks like my multipart implementation is almost done... I just need to loop it...
[12:20] * Dantman found a fun new way to do upload progress meters.
[12:22] <MisterN> yes?
[12:25] <Dantman> I found out that when you stream a html document back to the browser with script tags in the body the tags get evaluated as the client gets them...
[12:25] <Dantman> So if you make sure that you aren't using anything that will buffer responses, you can stream progress indications as script tags which will execute as they arrive.
[12:26] <ondras> have you ever seen cgi-irc? :))
[12:26] <ondras> it works *exactly* on this principle
[12:26] <Dantman> Heh, never looked at how it works...
[12:26] <Dantman> Though, isn't that one way?
[12:27] <Dantman> ;) I'm meaning that the very same request that the upload is being sent to the server with, can have script tags with progress indications streamed back as that request feeds data from the client.
[12:27] <Dantman> No extra ajax or websocket calls.
[12:28] <ondras> yes, exactly
[12:28] <ondras> cgi-irc starts loading a "document" which periodically outputs script nodes
[12:28] <ondras> these are used to notify client about irc activity
[12:28] <ondras> no ajax (xhr) needed :)
[12:30] <Dantman> And if you make use of a timeout checking readyState, domready, and a tail that clears the timeout you can even catch any abort to the upload made by the user or unexpected server failure.
[12:31] <Dantman> ^_^ I went a step further and made the iFrame use the parent frame's jQuery instance to fire events from the iframe to abstract the upload information.
[12:31] <Dantman> Heh, so my iframe node is firing kuploadprogress, kuploadabort, and kuploadfinished events.
[13:29] <Dantman> ^_^ I finally have a use for a MongoDB capped collection.
[13:30] <Dantman> upload data before making the upload a permanent file.
[14:33] <Dantman> Mmm... ^_^ creating a md5 digest of an upload from a file without buffering the entire file into ram was much more fun...
[14:34] <Dantman> I suppose I could do that while I'm uploading the file... But that would make the code ugly T_T
[15:15] <Dantman> Heh, file uploads could make for DoS potential (if multiple people upload files at the same time taking up all the workers there won't be workers to serve out content)...
[15:16] <Dantman> ^_^ But I might have some fun working around that with varnish... ;) ie: pipe file upload requests to an instance dedicated to file uploads... heh
[15:28] <Wes-> ondras: pong
[15:30] <ondras> Wes-: what happens in SM when you try to create a JS string from an invalid sequance of bytes?
[15:47] <Wes-> ondras: there is no such thing as an invalid sequence of bytes in JS Strings
[15:47] <Wes-> ondras: JS Strings are technically indistinguishable from an array of unsigned int16
[15:47] <Wes-> ondras: OTOH -- JS *source code* must be Unicode
[15:48] <Dantman> *sigh* I run into java and it's stupid signed bytes again... now where did I put that conversion code.
[15:49] <ondras> Wes-: okay - 197, 190, 197
[15:49] <ondras> Wes-: what happens if you create a string in SM from these 3 bytes, pretending it is utf8
[15:49] <Wes-> ondras: "pretending it's UTF8" is not relevant, you just create the string
[15:50] <Wes-> ondras: understanding UTF8 is not part of javascript
[15:50] <ondras> fine, so, what would the resulting string contain?
[15:50] <ondras> (looking from the JS side)
[15:50] <ondras> how many characters, which code points?
[15:51] <Wes-> ondras: e.g. var s = [ String.fromCharCode(197), String.fromCharCode(198), String.fromCharCode(197) ].join('')
[15:51] <Wes-> ondras: the string will contain the following bytes
[15:51] <Wes-> 0 197 0 190 0 197
[15:51] <Wes-> assuming big-endian architecture
[15:51] <ondras> aha
[15:52] <ondras> strings in V8 api are created from utf8
[15:52] <ondras> but there seems to be no error handling as far as I can see
[15:52] <Wes-> ondras: or UTF-16 IIRC
[15:52] <ondras> yeah, possibly
[15:52] <Wes-> ondras: but remember, creating the string and having the string are slightly different concepts
[15:53] <Wes-> ondras: to be crystal clear, STrings in javascript are arrays of uint16 -- converting them to something more meaningful for I/O or what have you is *convention*
[15:53] <Wes-> ondras: Also, you can (believe it or not) have valid Unicode Strings which are composed of invalid Unicode sequences
[15:54] <Wes-> Now, what your I/O routines are supposed to do with invalid Unicode sequences is not specified IIRC
[15:55] <ondras> #!v8cgi
[15:55] <ondras> var Buffer = require("binary-f").Buffer;
[15:55] <ondras> var b = new Buffer("???", "utf-8");
[15:55] <ondras> var s = b.toString("utf-8", 0, 3);
[15:55] <ondras> system.stdout(s + "\n");
[15:55] <ondras> system.stdout(s.length + "\n");
[15:55] <ondras> this is the case
[15:55] <ondras> I am talking about
[15:56] <ondras> and the output:
[15:56] <ondras> ??
[15:56] <ondras> 2
[15:56] <Wes-> ondras: What code points are "???" ?
[15:57] <ondras> 382, 353, 269
[15:57] <Dantman> T_T I can't remember where I put that code to convert between signed and unsigned numbers
[15:57] <Wes-> okay
[15:57] <ondras> and s.charCodeAt(1) is 65533
[15:58] <MisterN> oh right czech does use these weird characters
[15:58] <Wes-> I think you are confused about what you are asking your code to do
[15:58] <MisterN> that's where i had seen them before
[15:58] <ondras> so, I am giving V8 three bytes, telling him they are utf8 (which is not true), but no exception is thrown
[15:58] <Wes-> No, you are giving V8 three uint16s
[15:58] <Wes-> you can't even fit 382 in a byte
[15:59] <ondras> who says I want to do that?
[15:59] <Wes-> ondras: "So I am giving v8 three bytes"
[15:59] <ondras> Wes-: yes, that is exactly what I do
[15:59] <Wes-> you said code points for "???" is 382,353,269
[15:59] <Wes-> that's six bytes
[15:59] <ondras> yes.
[15:59] <ondras> I take 3 first bytes
[16:00] <ondras> 16:59 < ondras> var s = b.toString("utf-8", 0, 3);
[16:00] <Wes-> So, the ? and the first half of the ?
[16:00] <ondras> yep
[16:00] <ondras> http://bespin.cz/~ondras/html/classv8_1_1String.html#66188e5be9378cad8cbf953053de86bc
[16:01] <MisterN> completely arbitrary and undescriptive parameters to toString
[16:01] <MisterN> that's how i love it.
[16:01] <Wes-> And then you are instructing v8 to convert those three bytes from utf16 to utf8?
[16:01] <Wes-> sorry, not V8, your buffer class
[16:01] <ondras> Wes-: instructing v8 to treat these three bytes as utf8, which is not correct
[16:02] <ondras> the Buffer stuff around is not that important
[16:02] <Wes-> I don't understand how your are instructing v8 to do anything
[16:02] <Wes-> What is your buffer constructor doing?
[16:02] <Wes-> No, what the buffer stuff around it is critical
[16:02] <ondras> forget the buffer!
[16:02] <ondras> :)))
[16:02] <Wes-> utf8 is not part of jasvascript
[16:02] <Wes-> If you forget the buffer, then there is no unicode issue
[16:02] <Wes-> Strings do not contain unicode
[16:02] <Wes-> Strings contain arrays of unsigned 16-bit integers
[16:03] <ondras> v8::String::New("\197\190\197")
[16:03] <ondras> thisis what I do
[16:03] <ondras> forget buffer
[16:03] <Wes-> ...and?
[16:03] <ondras> http://bespin.cz/~ondras/html/classv8_1_1String.html#66188e5be9378cad8cbf953053de86bc
[16:03] <ondras> Allocates a new string from either utf-8 encoded or ascii data.
[16:03] <Wes-> But that interface is not part of javascript
[16:04] <ondras> noone said it is
[16:04] <Wes-> the only thing that is javascript is the resultant string
[16:04] <Wes-> So what are you expecting to happen, and what happens?
[16:04] <ondras> what happens is what I have shown
[16:04] <ondras> a 2-char js string is created
[16:04] <ondras> and no error is reported
[16:04] <ondras> which is, imho, weird
[16:04] <ondras> so I was curious if there is something similar in SM
[16:05] <Wes-> Well, maybe V8 assumes that if you are using the UTF8 interface that you will feed it UTF8 data
[16:05] <ondras> that is certain :)
[16:05] <Wes-> ondras: No, spidermonkey does not have UTF8 strings
[16:05] <ondras> okay :)
[16:05] <Wes-> Now, you can tell spidermonkey that C strings are UTF8
[16:06] <Wes-> and then ask it to convert some strings which aren't UTF8 into JS Strings as UTF16
[16:06] <Wes-> But, if they are in fact NOT UTF8, you will either get assertions or strange behaviour
[16:06] <Wes-> depending on if you have a debug or a release build
[16:06] <Wes-> But - this is on the C side only -- it is not exposed to script
[16:06] <Wes-> And as with anything in C -- garbage in, garbage out
[16:07] <ondras> MisterN: feel free to read the Binary/F spec to understand those
[16:07] <Wes-> But once you have your data into a JS String -- it is indistinguishable from uint16[]
[16:07] <Dantman> Aaahhh, there it is
[16:07] <Wes-> Whoop!
[16:07] <Dantman> Thank you fgrep
[16:08] <fgrep> Dantman: you're welcome
[16:09] <Dantman> heh
[16:09] <ondras> Wes-: if you checked that code with Buffer
[16:09] <ondras> you see what I do there
[16:09] <ondras> (if you already glanced at B/F proposal)
[16:09] <Wes-> ondras: Haven't had time to, unfortunately
[16:09] <Wes-> well, I've *glanced* at it
[16:09] <ondras> I take first 3 bytes from a string encoded as utf8 and try to create a JS string
[16:10] <ondras> now these bytes do not form a valid utf8
[16:10] <Wes-> There is no such thing in script as a string encoded as utf8
[16:10] <ondras> but I have no means (with V8 api) to detect error
[16:10] <Wes-> But yes, of course, you can create buffers which do not contain valid unicode
[16:10] <ondras> string encoded in buffer as utf8
[16:10] <MisterN> the Binary/* situation is not satisfactory.
[16:10] <Wes-> ondras: how do you handle other character sets?
[16:10] <ondras> I am not a complete unicode idiot, please :)
[16:11] <MisterN> Binary/F does not really convince me. it's confusing and seems over-simplified.
[16:11] <ondras> Wes-: iconv.. but when the toString specifies utf8, I originally thought iconv is not necessary
[16:11] <MisterN> the function descriptions are hard to understand.
[16:11] <Wes-> ondras: If iconv is not used, then you will need to handle the error cases yourself
[16:11] <ondras> yeah, unfortunately
[16:11] <Wes-> ondras: v8 would be insane to error check those in a release build
[16:11] <MisterN> what's the difference between toString and read?
[16:12] <ondras> or I can use iconv to do utf8->utf8 conversion
[16:12] <MisterN> is there something to convert between to different charsets?
[16:12] <ondras> to check for errors
[16:12] <Wes-> ondras: since the only way to get a UTF8 string is from C
[16:12] <ashb> i also dont think the binary proposal should have particla encoding support
[16:12] <ashb> cos its broken
[16:12] <Wes-> ondras: No point in doing that IMO, just give iconv the UTF16 string and let it do the transcode to utf8
[16:12] <MisterN> ashb: "particla"?
[16:12] <Wes-> ondras: Or write the error-scanning code yourself, it's not terribly hard
[16:12] <ondras> Wes-: which utf16 string? :)
[16:13] <Wes-> ashb: I think there is room for handling *unicode* encodings in there
[16:13] <ashb> MisterN: particularly
[16:13] <ondras> Wes-: as you see, the buffer was created by encoding unicode points in utf8
[16:13] <MisterN> ashb: then i still don't understand what you mean.
[16:13] <MisterN> ondras: on a completely unrelated note, breznak is good beer.
[16:13] <ondras> MisterN: agreed!
[16:13] <ondras> b?ez??k
[16:13] <ondras> more unicode!
[16:14] <MisterN> heh i could type it, if i could remember where which sign belongs :D
[16:14] <MisterN> no i actually can't type it :(
[16:15] <Wes-> ondras: No, the buffer was created by lying to a binary-level interface about what your bytes were
[16:15] <MisterN> i have only skimmed some of the proposals, but i still like Binary/B the best.
[16:16] <Wes-> ondras: To create a buffer of UTF8 from a String, you should be doing a UTF16->UTF8 conversion
[16:16] <MisterN> one big advantage is that we have already implemented it :D
[16:16] <Wes-> MisterN: I will probably rename my binary binary-b FWIW
[16:17] <Wes-> MisterN: Then make binary load binary-whatever
[16:17] <MisterN> Wes-: hmm.
[16:17] <MisterN> Wes-: and make your existing code just use binary-b?
[16:17] <Wes-> MisterN: yeah
[16:19] <Wes-> I am actually considering implementing binary-f with binary-b ;)
[16:22] <ondras> Wes-: you are wrong - the buffer was created by calling V8 api which provides a utf8 representation of js string
[16:23] <ondras> Wes-: I still cannot get rid of a feeling that you somehow treat me like an incompetent dumbass, when it comes to unicode :)
[16:23] <Wes-> ondras: Sorry, not meaning to make you feel like that
[16:23] <ashb> ondras: very few people actually get unicode mind
[16:23] <Wes-> ondras: But I am trying to understand what you are considering to be a problem
[16:24] <ondras> no offense taken, the problem most probably lies in my inability to express myself in english
[16:24] <ondras> but also note that V8 api might not be a standard one
[16:24] <Wes-> ondras: What you are saying is that your string is already UTF8 instead of UTF16 as expected (v8 optimization), so then your conversion operation should just be a nop
[16:24] <ondras> (for instance, as I already mentioned, it provides a utf8 representation of a js string)
[16:24] <ondras> Wes-: precisely
[16:25] <Wes-> ondras: So then you are taking part of that utf8 and converting to utf8 again?
[16:25] <ondras> yes
[16:25] <ondras> http://www.reddit.com/r/javascript/comments/b6uvc/obfuscate_and_execute_javascript_without_any/
[16:25] <Wes-> And what are you expecting to happen?
[16:26] <ondras> not sure :)
[16:26] <Dantman> ^_^ YAY... I have file uploads... multipart parsing complete... file copy complete... and so much more complete...
[16:26] <Dantman> Now I can commit and go to bed...
[16:27] <Wes-> ondras: Does Binary/F say you have to throw on an invalid string?
[16:27] <Wes-> s/string/buffer
[16:27] <Wes-> or, inability to convert the buffer, rather?
[16:27] <ondras> yes
[16:27] <ondras> Throws a RangeError if the buffer is malformed for the given character set, if a multi-byte character would be split across the stop boundary, or if any of the code points are out of the implementation's supported range.
[16:28] <ashb> range error for malformed seems wrong ther emind
[16:29] <ondras> Wes-: but I can handle that (as said before, either by checking myself or doing 1:1 iconv)
[16:30] <Wes-> ondras: In that case, to meet the binary-f spec, I think you should either change interfaces or implement error checking on the input buffer
[16:30] <Wes-> It's actually not that hard to do it yourself
[16:30] <Wes-> the base alrogirhtm is
[16:30] <Wes-> :loop
[16:30] <Wes-> look at character?
[16:31] <Wes-> depending on value, this is a 1, 2, or 3 byte code point
[16:31] <Wes-> n = 1, 2, or 3
[16:31] <Wes-> check next n-1 characters to make sure they are less than some specific value
[16:31] <Wes-> set character pointer to here + n
[16:32] <Wes-> goto loop
[16:32] <Wes-> the details are in the unicode spec FWIW
[16:32] <ondras> I already have a utf8 encoding impl in JS (for different project)
[16:32] <ondras> so no prob here
[16:32] <ondras> anyway, thanks for consultations
[16:33] * ondras has to leave now for a dinner
[16:33] <Wes-> enjoy! :)
[17:23] <MisterN> Wes-: what do you use for ffi? i just stumbled upon this: http://sourceware.org/libffi/
[17:23] <MisterN> Wes-: or did you write the entire ffi yourself?
[17:25] <ashb> no he uses libffi i think
[17:25] <Wes-> MisterN: That's the ffi that everybody uses, e.g. python ctypes, even gcc itself
[17:26] <Wes-> MisterN: All that library does, though, is abstract away the calling conventions for the current ABI
[17:26] <ashb> MisterN: i was thinking about writing some glue so that we can just steal his ffi module as wholesale
[17:26] <Wes-> i.e. on sparc you pass arguments in registers, on intel you pass them on the stack
[17:26] <Wes-> ashb: the glue should be pretty easy
[17:26] <ashb> Wes-: for the most part. binary makes it a little bit more complex
[17:27] <Wes-> ashb: Yeah, although you could just not support that. (which would kind of suck)
[17:27] <ashb> more than kind of :)
[17:27] <ashb> but for started, its true we could not
[17:27] <ashb> then we still have access to things like fork and what not
[17:27] <ashb> without having to write lots of short C++ methods/modules
[17:27] * Wes- nods
[17:27] <MisterN> ashb: we could also embed the sourceware.org lib
[17:28] <Wes-> you can also naively convert to string with the gffi Memory.prototype.asString() method
[17:28] <ashb> MisterN: there's more to it than that
[17:28] <ashb> and wes has worked out most/all of hte issues already
[17:28] <Wes-> MisterN: That gains you almost nothing
[17:28] <MisterN> why? they handle the problem that requires you to write platform-dependent code
[17:29] <Wes-> ashb: make sure you steal the makefile too, that's where all the header smarts are
[17:29] <Wes-> MisterN: Go ahead then. ;)
[17:29] <Wes-> MisterN: All libffi does is allow you to pass arguments to function pointers
[17:29] <ashb> Wes-: i only said thought about. At the momenti'm unlikely to do anything
[17:29] <MisterN> heh maybe invoking g++ on demand would also be possible :D
[17:29] <Wes-> that's the easy part
[17:30] <MisterN> Wes-: it's the easy part? huh? what else is problematic?
[17:30] <ashb> GC
[17:30] <Wes-> MisterN: the hard part is putting the argument lists together (type coerscion), getting the function pointers, and header values
[17:30] <ashb> argument marshalling
[17:30] <Wes-> Yeah, GC finalization is tricky too
[17:31] <Wes-> getting function pointers is also much trickier than you would think
[17:31] <MisterN> ah the kind of stuff we handle at compile-time
[17:31] <ashb> yup
[17:31] <MisterN> Wes-: dlsym is tricky? :P
[17:31] <Wes-> Mistern: quick, what is the dlsym() argument you pass to get fstat on linux?
[17:31] <Wes-> hint: the answer is not "fstat"
[17:31] <MisterN> ah teh macro joy
[17:32] <Wes-> right
[17:32] <ashb> like i said - WQes has sorted out most/all of the problems
[17:32] <ashb> it also means we dont have to maintain the code
[17:32] <Wes-> so I have provided trampolines for every function defined by whatever posix version you are compiling as
[17:32] <ashb> s/Q//
[17:32] <ashb> i'm all in favour of someone else maintaining scarry code
[17:33] <Wes-> scarry, yeah, it did leave scars in my brain!
[17:33] <ashb> >_>
[17:33] <ashb> hmmmm pdoc time!
[17:33] <ashb> TDD ftw
[17:33] <Wes-> You are deaf?
[17:33] <ashb> no?
[17:33] <MisterN> Wes-: irc doesn't transmit sound very well
[17:34] <Wes-> ashb: future-looking note -- gffi will probably lose X.call() to be replaced by X() -- but .call() will be lazily grandfathered if possible
[17:34] <Wes-> ashb: Then what are you doing with a TDD?
[17:34] <ashb> test driven development?
[17:34] <Wes-> ashb: http://en.wikipedia.org/wiki/Telecommunications_device_for_the_deaf
[17:35] <Wes-> My interpretation pre-dates yours! :)
[17:35] <ashb> i was googling working out what else tdd might mean
[17:35] <ashb> its no where in the first few pages
[17:35] <Wes-> I remember trying to hack into a TDD service with a VIC-20 modem
[17:35] <Wes-> tricky, they use a different frequency for FSK and BAUDOT instead of ASCII
[17:35] <Wes-> I guess deaf people have poor pagerank
[17:35] <MisterN> oO
[17:37] * Wes- is sad
[17:37] <Wes-> I am re-writing a version of iconv() that dlopens libiconv()
[17:38] <Wes-> This is the height of stupidity
[17:38] <Wes-> How many freaking versions of iconv do we need?!!
[17:38] <ashb> why are you doing that?
[17:38] <Wes-> I can't reliably link against a specific one because the linker syntax does not allow absolute paths
[17:39] <Wes-> And I can't tweak -L / -R to do what I need, because then I get the wrong libstdc++
[17:39] <ashb> you can't?
[17:39] <ashb> i thought you could just specify /path/to/libiconv.so ?
[17:39] <MisterN> Wes-: oh we have the same problem!
[17:39] <ashb> you can with gcc-ld anyway
[17:39] <Wes-> No, that only works for object archives
[17:40] <MisterN> Wes-: but you should use the glibc iconv on linux
[17:40] <ashb> Wes-: are you sure?
[17:40] <Wes-> MisterN: Right, but I want solaris iconv on solaris apple iconv on mac
[17:40] <Wes-> ashb: pretty sure
[17:40] <ashb> try :)
[17:40] <ashb> i'm fairly sure thats what cmake does for a lot of its linkings
[17:41] <ashb> /usr/local/lib/libboost_unit_test_framework-mt.dylib ../lib/libflusspferd.dylib
[17:41] <ashb> ^^ part of the link command generated by cmake for instance
[17:42] <Wes-> ashb: Huh, seems to work on mac
[17:42] <MisterN> ashb: cmake strips the rpath on install
[17:42] <ashb> MisterN: *can* strip the rpath
[17:42] <Wes-> ashb: does that also force it to use that one at run-time?
[17:43] <MisterN> ashb: i think it normally does
[17:43] <ashb> MisterN: yeah, you can turn it off
[17:43] <ashb> Wes-: i think it depends on the libdl in use
[17:43] <MisterN> well i guess that improves matters
[17:43] <ashb> i.e. -Wl,-soname=/foo/bar
[17:43] <ashb> will be used in over what is actually specified as the link name
[17:44] <ashb> but most of the time the soname is the same as the file name
[17:44] <Wes-> hmm, now I need to see if that will give me grief with the solaris linkk
[17:44] <Wes-> wait, -Wl-soname, in the -L library affects what we look for at runtime?
[17:44] <ashb> yes
[17:44] <ashb> its a fucking pain
[17:45] <Wes-> God I hate linkers
[17:45] <ashb> i.e. i was renaming the libmozjs.so to libmozjs-1.9.2.so
[17:45] <ashb> via mv
[17:45] <ashb> and at runtime it still looked for libmozjs.so
[17:45] <ashb> cos of the -soname
[17:45] * Wes- growls
[17:45] <ashb> yeah took me a while to work out wtf was going on there
[17:45] <Wes-> I wouldn't be so bad if the whole world copied what apple was doing
[17:46] <ashb> @exectuable_name/ ?
[17:46] <Wes-> Yeah, they can also specific @rpath and a bunch of other things
[17:46] <ashb> $ORIGIN/
[17:46] <ashb> works for linux ld
[17:46] <ashb> (i think its that)
[17:46] <ashb> there's also a runpath which behaves better than rpath
[17:46] <Wes-> yeah - but I don't like the fragmentation
[17:46] <Wes-> everybody should pick a way and do that!
[17:46] <ashb> http://linux.die.net/man/1/chrpath
[17:47] <ashb> yeah
[17:47] <Wes-> It's not like compiler options are brand differentiators
[17:47] <ashb> as far as i can work out runpath gets round all the problems i've read about rpath
[17:47] <Wes-> OOoooh, I'm going to buy an apple because it supports @executable_name !
[17:48] <MisterN> Wes-: tell all your friends to do the same!
[17:48] <Wes-> on the plus side, lazy-loading libiconv means I don't have to yank it in for the binary module until you actually use it
[17:48] * Wes- is counting the saved nanoseconds already
[17:48] <MisterN> i think it's even on the order of tens of microseconds
[17:49] <Wes-> BTW did you guys notice how fast ondras whipped together Binary/F
[17:49] <Wes-> I'm kinda jealous :)
[17:55] <ashb> there's not all that much in F mind
[17:55] <Wes-> true
[17:56] <Wes-> I was thinking of doing F as my first fully COW module, but then I noticed that range wasn't a copy
[17:56] <ashb> its not?
[17:56] <ashb> i htought that was the entire point of it?
[17:56] <ashb> you can always do it as CoW anyway, no?
[17:56] <Wes-> No, I think it's a view
[17:56] <ashb> oh
[17:56] <ashb> yeah it might be
[17:57] <ashb> ... what ever *for*?
[17:57] <Wes-> ashb: *guessing* structs
[17:57] <ashb> (x.range(5))[5] === x[10] ?
[17:57] <ashb> ondras: ^^ ?
[17:58] <Wes-> more like memcpy((char *)s + offsetof(struct pwent, pw_nam), "wes");
[17:58] <Wes-> except in highly unportable javascript
[17:59] <Wes-> (gpsee FFI structs are intended to be completely portable across POSIX FWIW -- still need to iron out details for 3rd party lib structs)
[17:59] <ashb> also for added fun i'll want to support windows
[18:00] <Wes-> ashb: That might be more than you want to bite off
[18:00] <ashb> not in terms of psoxi apis, obv
[18:00] <Wes-> ashb: Well, maybe not
[18:00] <ashb> *posix
[18:00] <ashb> just ffi working
[18:00] <ashb> i think most of the windows API is stdcall
[18:01] <Wes-> ashb: The tricky part will be parsing the windows header files without being able to use GCC
[18:01] <ashb> when do you do the parsing?
[18:01] <ashb> at module compile time?
[18:01] <Wes-> ashb: If you *do* head towards windows, I would be willing to tweak gffi to understand stdcall/cdecl
[18:01] <Wes-> ashb: yes
[18:02] <ashb> well 1( it could just not parse
[18:02] <Wes-> ashb: OTOH if you windows libs have .tlbs we could make use those instead
[18:02] <ashb> or 2) at the moment we only support mingw
[18:02] <ashb> we = me cos no one else really gives a shit about windows
[18:02] <Wes-> ashb: Third option, you could build libffi with mingw/cygwin and everything else with VC
[18:03] <ashb> libffi is probably more likely to compile with VC than the rest of libflusspfer
[18:03] <Wes-> that would have the strange side effect of making all the POSIX stuff work, too, but it would call into cygwin.dll for them
[18:03] <ashb> last time we tried it didn't like some of the template stuff MrN was doing :)
[18:03] <ashb> (or something like that)
[18:03] <Wes-> ah, yes
[18:03] <Wes-> I can see that
[18:03] <MisterN> ashb: i do give a shit about windows
[18:03] <ashb> I dont like cygwin
[18:03] <MisterN> not so much about ffi for windows tho
[18:03] <ashb> MisterN: kk. most other commonjs people dont tho
[18:03] <Wes-> ashb: does mingw expose a full posix api?
[18:03] <MisterN> well i don't want to use windows either
[18:04] <ashb> Wes-: cygwin to my mind is like X11 of OSX. it works, it just feels alien and horrible
[18:04] <MisterN> but for flusspferd to succeed, we need windows support :D
[18:04] <ashb> Wes-: no its just a gcc compiler toolchain
[18:04] <MisterN> ashb: so essentially i want windows support to boost my ego. :D
[18:04] <ashb> MisterN: yeah, and where at all possible I'd like the same level of module support on all platforms
[18:04] <ashb> MisterN: best of reasonds :D
[18:04] <Wes-> ashb: Right -- if you compiled with cygwin and linked with cygwin.dll, you would be able to have totally portable JS code that called into FFI
[18:05] <ashb> eh :)
[18:05] <ashb> thats once we have ffi working
[18:05] <Wes-> well, yeah. :)
[18:06] * ashb goes back to working on PDoc some more
[18:06] <Wes-> FWIW I might be willing to port gffi as far as cygwin, including specifiable calling conventions
[18:07] <MisterN> Wes-: and mingw?
[18:07] <MisterN> Wes-: ashb uses mingw
[18:07] <ashb> it looks like libffi might already support it
[18:07] <Wes-> MisterN: Right, he would have to go the last mile on the gffi code. libffi should be fine anywhere, it's used internally by gcc
[18:07] <MisterN> i thought libffi doesn't do everything you need?
[18:08] <ashb> it doesn't. but it handles all the calling convetion stuff i think
[18:09] <ashb> the rest of gffi is i guess header parsing to automatically do type coercions?
[18:09] <ashb> i.e. you dont have to specify param types/returns for posix fns?
[18:09] <MisterN> with automatically generated regexes if i understood Wes- correctly
[18:09] <Wes-> ashb: No, calling convention has to be specified on a function by function basis to libffi
[18:09] <MisterN> which is... scarry
[18:09] <MisterN> scarry scarry stuff :D
[18:09] <MisterN> (yes the double r is intentional :P)
[18:10] <ashb> Wes-: sure, but its just 'its this type' and you dont have to manually deal with it more than that
[18:10] <Wes-> ashb: type coersion isn't done by parsing headers, it's done by looking at argument types in the ffi function declaration
[18:10] <ashb> Wes-: ah
[18:10] <Wes-> ashb: header parsing is used so that the JS programmer can say O_RDWR without having to know that that is 6 on linux and 3 on windows
[18:11] <Wes-> we also use some compile-time tricks to get integer rangers right and so forth for types
[18:11] <ashb> cool
[18:11] <Wes-> for example, if you specific a -1 into a size_t argument on the js side, we will actually (size_t)-1 that on the C side
[18:11] <ashb> MisterN: that reminds me - we should probably expose overload of operator == to our objects
[18:12] <MisterN> ashb: ?
[18:12] <Wes-> which is actually important, because the result could be different when changing integer sizes -- and when JS is using jsdouble *
[18:12] <ashb> spidermonkey lets you provide custom == (but not ===) for native objects
[18:12] <ashb> a field in JSClass
[18:12] <Wes-> no, JSExtendedClass
[18:12] <ashb> that one
[18:14] <MisterN> ashb: hmm you think we should support it?
[18:14] <ashb> dunno. depends if v8 or JSC support it
[18:14] <ashb> they probably dont
[18:16] <Wes-> considering javascript does not support that -- I wouldn't hold my breath
[18:16] <ashb> yeah
[20:19] <ondras> ashb: yes
[20:19] <ashb> CoW or not?
[20:20] <ondras> wtf CoW
[20:20] <ashb> Copy-on-write
[20:20] <ashb> does range create a view, or a new buffer?
[20:20] <ondras> view
[20:21] <ashb> if its just a view, what is the point of it?
[20:21] <ondras> not sure
[20:21] <ondras> I already asked Kris
[20:21] <ashb> since isn't it just [x+5] ?
[20:21] <ondras> sec, will give a link
[20:22] <ashb> k
[20:22] <ondras> http://groups.google.com/group/commonjs/browse_thread/thread/db5a87e833f07a3f
[20:22] <ondras> how to permalink a post there...
[20:22] <ondras> just search for message ~#5, my first there
[20:22] <ashb> yeah i've never worked that out
[20:22] <ashb> got it
[20:23] <ashb> oh so, the use is for shrinking the length.
[20:23] <ashb> hmmm
[20:23] <ondras> actually, the whole "view" feature remains mysterious for me
[20:23] <ashb> yeah
[20:23] <ondras> but it is a one-object-only binary proposal
[20:24] <ondras> and that means I implement it :)
[20:24] <ashb> heh
[20:27] <ashb> ondras: okay so the use case is
[20:28] <ashb> bigBlob = new Buffer(16*1024); n = 0; l = readInto(bigBlob, n, ..); callback( bigBlob.range(n, n+l) );
[20:29] <ashb> i.e. allocate a big chunk of memory and just use it up as you go
[20:34] <ondras> hm
[20:34] <ashb> i.e. its a way to use a buffer as a memory pool
[20:34] * ondras still thinks this could/should be done optionally at js level as a helper
[20:35] <ashb> perhaps, yes. but you do want that helper to look like a blob too
[20:35] <ondras> yes
[20:36] <ondras> (that's what we have inheritance for)
[20:36] <ashb> typically inheritance to a native object is tricky
[20:36] <ondras> true
[20:36] <ondras> but I tried that with V8 and Array
[20:36] <ashb> also ryan is worried about speed. (needlessly slow sometimes in my view)
[20:36] <ondras> when I experimented with Binary/Lite
[20:36] <ondras> and it worked just fine
[20:36] <ashb> *so
[20:37] <ondras> I am still not sure if ryan's speed needs aren't best solved without javascript
[20:39] <MisterN> if you need obscene speed, you can always use c++.
[21:00] <Wes-> Mistern: Or C
[21:00] <Wes-> ;)
[21:00] <ondras> :P
[21:00] <Wes-> But, seriously, he wants to do fast things with javascript, specifying interfaces to make that possible doesn't seem like a problem to me
[21:01] <Wes-> even spidermonkey does that
[21:01] <MisterN> Wes-: yeah sure, making it possible to be fast in js is definitely the way to go
[21:01] <Wes-> for example, var s = "a" + "b" + "c" does not instanciate 5 strings like the statement impleies
[21:01] <MisterN> i mean i insisted on the same thing in the encodings module :P
[21:01] <MisterN> :P
[21:07] <hannesw> i think the speed thing is a bit exaggerated
[21:07] <hannesw> with a generational GC like sun's hotspot, you can literally create millions of arrays per second and it's hardly measurable
[21:08] <hannesw> and i don't think v8's GC can be much worse than sun's
[21:09] <Wes-> hannesw: Is v8's GC even generational? Spidermonkey's isn't
[21:09] <ashb> inimino: bad monkey. no cookie: Array.prototype.uniq=function(){
[21:09] <ashb> Wes-: yes, v8's is
[21:09] <Wes-> nice
[21:09] <ashb> your objects will move about underneath you
[21:09] <Wes-> not nice
[21:10] <ashb> so you have to use Handle<array> etc
[21:10] <ashb> i.e. its a managed pointer
[21:10] <ashb> you just can't rely on the address. it practice it doesn't move
[21:10] <hannesw> i don't know, but V8 rivals the JVM and C++ speed-wise
[21:10] <MisterN> btw do you know this? http://heroku.com/
[21:11] <ashb> yeah its a really cool idea
[21:11] <MisterN> except for the gradient in the big number, i also like the design :D
[21:11] <MisterN> gradients in text :(
[21:11] <ashb> that's kind of what smart platforms is going for
[21:11] <ashb> (a git repo you can push code to etc)
[21:12] <MisterN> ultimately it's all stealing the idea from google :D
[21:17] <inimino> ashb: heh, you don't like my uniq? :)
[21:31] <ashb> inimino: you dont even use it
[21:32] <inimino> ashb: ah, you're right
[21:32] <ashb> it was confusing me, cos i was only including the PEG_generator to debug something quickly with showTree
[21:33] <inimino> ashb: yeah, it's included in a file that's used in several other places, but it really doesn't need to be there
[21:33] * inimino makes a note
[21:34] <inimino> ashb: you should just be able to include PEG_util.js for that stuff
[21:35] <ashb> ah yeah
[21:35] <ashb> dont need it most of the time
[21:35] <ashb> just while i was working on the tree walkers
[22:45] <ashb> inimino: is it possible to get the PEG to emmit events instead of a parse tree?

 

 

Logs by date :