Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2010-03-02:

[0:00] <ryah> Wes-_: i don't necessarily do much utf8 - i do do lots of ascii
[0:00] <ryah> and for ascii i do use the Buffer's buffer for the string
[0:01] <Wes-_> ryah - interesting, why not much utf8? You don't process user-content heavily?
[0:01] <ryah> Wes-_: well, i don't optimize for that
[0:02] <Wes-_> Makes sense. Everything I do has to be minimum iso-8859-1 (french canadians), so I have been doing lots of THAT since the late 90s, but have been slowing moving over all my content to utf-8 for a variety of reasons... Including Chinese people and bugs in Apple's terminal
[0:03] <Wes-_> Does node.js have something like a ssjs content markup? With gettext()-like interfaces? Or is the focus still more app oriented?
[0:04] <Wes-_> kriskowal: thanks (sort of ;) ) for asking me to put that together, it really made me think about how to actually accomplish that goal and cleared up a few bad mental pathways
[0:06] <ryah> Wes-_: nope
[0:10] <kriskowal> Wes-_ similarly, i did some homework that has flushed some bad headspace. i implemented iconv bindings for node+narwhal this weekend.
[0:11] <kriskowal> i'm convinced to put off crazy transcoding triangulation ideas and just spec an API for iconv
[0:11] <kriskowal> and having done that, i have much better footing to consider your proposal, and to consider how to cut this
[0:12] <kriskowal> we definitely need to draw a line somewhere between the encoding api (which would wrap iconv) and the buffer api (which would wrap basic encodings supported by the engine)
[0:13] <kriskowal> the implementation i threw together this weekend supports ascii and utf8 at the embedding layer, because that's what v8 gives you.
[0:14] <kriskowal> then i created a Transcoder API based on iconv.
[0:14] <kriskowal> i split the javascript Buffer implementation between the embedding stuff with "node/buffer-engine", then created a pure-js "node/buffer" that monkey patches the methods to add support for arbitrary transcoding
[0:15] <kriskowal> my next step is to implement transcoding streams in terms of hannes's IO/A the transcoder
[0:16] <kriskowal> from there i'll have a better idea about how to evaluate your proposal
[0:16] <kriskowal> Wes-_^
[0:17] <kriskowal> i also suspect that there's something wrong with node/v8 encoding because it's silently stripping narwhal's vt100ness
[0:18] <kriskowal> could be stdio/stderr synchronization though; so the jury is still out
[0:21] <Wes-_> kriskowal: the Encodings API from MisterN is essentially an iconv wrapper, and it has a working implemenation
[0:22] <kriskowal> yeah, my impl is based on it loosely
[0:22] <kriskowal> which isn't to say it won't eventually be exactly
[0:22] <kriskowal> just still exploring
[0:22] <MisterN> i guess if Binary/X becomes Binary at some point we'll have to change the Encodings API, too.
[0:23] <kriskowal> yeah
[0:23] <MisterN> i don't see any other fault with it :P
[0:23] <Wes-_> stripping vt100ness, that's interesting. If you are outputting esc codes on stderr and text on stdout (or vice-versa) you can bump into funny issues -- stderr flushes immediately on write, stdout does not flush until \n
[0:23] <kriskowal> yeah, we use explicit flushes to sync
[0:23] <kriskowal> it could be a problem with my flush bindings
[0:24] <Wes-_> you could potentially test with 2>&1
[0:24] <Wes-_> although I'm not 100% that will work
[0:24] <kriskowal> i did a quick test altering our term module to send both to stdout and the problem remained
[0:25] <kriskowal> which gives me reason to suspect that the problem is elsewhere, but i still need to trace through the dataflow to make sure something's not getting lost in transit
[0:25] <kriskowal> and, indeed, duping err to out has no effect
[10:22] <Dantman> 3s total, about 1s a request... to 1.5s total about 500ms a request... ^_^ Sweeet... I just made my http server respond to http requests much faster by using a blocking queue.
[10:24] <ashb> 3s request time is quite slow for most cases, isn't it?
[10:25] <Dantman> ashb, Total... as in all 7 http requests plus render time.
[10:25] <ashb> ah
[10:25] <Dantman> ^_^ So I basically cut page load time by half.
[10:26] <Dantman> Just by eliminating the delay between accepting a socket connection, and the time when a http worker claims that connection.
[10:26] <evilstreak> 7 http requests?
[10:26] <evilstreak> can't cut that down further?
[10:27] <Dantman> Not without making unnecessary reloading of data.
[10:27] <ashb> is there more than 1 JS or CSS file?
[10:27] <ashb> more than one sprite image?
[10:28] <Dantman> page's request + global.css + system.css + ui.css + allinone.js + ui.js + sitemap.js
[10:28] <Dantman> Everything except sitemap.js is a file built by combining a bunch of files together automatically (^_^ And with a bit of css transcoding fun to).
[10:29] <Dantman> ;) allinone.js alone turns what would be 13 http calls into one.
[10:30] <ashb> good good
[10:30] <Dantman> It's not going to be much of a problem in production. Right now I have cache: true, but expires: false because I'm developing and don't want the browser to go caching the code giving me trouble.
[10:31] <Dantman> In production all of those files will have a 30day expiry time.
[10:31] <Dantman> ;) So normally that page request will be the only one done.
[10:32] <Dantman> Heh, I might want to do something about the 175kb size of allinone.js though... *snicker*
[10:33] <Dantman> ^_^ At least I eliminated ckeditor from allinone or editor... that made them exponentially bigger.
[10:35] <ashb> typically i'd cache them indevenitly and change a version number
[10:35] <Dantman> Mhmm... I'll probably do that in production.
[10:36] <Dantman> Perhaps I'll make use of the git commit id for that purpose. And leave it off in dev.
[10:36] <evilstreak> 175kb? ouch
[10:36] <evilstreak> what's the CSS file at?
[10:37] <Dantman> evilstreak, non-gzipped though, I haven't built that in yet, on our actual server apache is doing the gzipping. I'm doing this locally so I don't have anything in between me and Kommonwealth.
[10:37] <Dantman> Hosted sites aren't a problem, I'm already planning to use the snapshot id as a cache break.
[10:37] <evilstreak> minified though?
[10:37] <Dantman> Nope, haven't built that in either
[10:37] <evilstreak> ah
[10:38] <evilstreak> not so bad then
[10:38] <Dantman> So ya, I can eliminate a fair bit of size by dynamically minifying and gzipping them
[10:39] <evilstreak> we have 264k of JS, after minification
[10:39] <evilstreak> it makes kittens cry
[10:39] <Dantman> CSS files are at 1kb, 1kb, 6kb, 0kb, and and 36kb
[10:39] <evilstreak> tiny then
[10:39] <Dantman> The 36kb one is ui.css which is that size because jQuery.UI's css is present.
[10:40] <Dantman> evilstreak, I think I have something that'll make you feel better
[10:40] <evilstreak> we have some disgusting amount of CSS too. IE6 groans under the effort of parsing the CSS and JS (parsing, not running, just parsing)
[10:41] <Dantman> evilstreak, http://eckharttolle.com/allinone.min.js
[10:42] <evilstreak> that's fairly epic
[10:42] <Dantman> ^_^ That's the majority of eckhartolle's bandwidth...
[10:43] <Dantman> *sigh* Kommonwealth v2 had some issues...
[10:46] <Dantman> evilstreak, Unminified allinone.js on kv2 was 13424 lines in size... Not to mention the fact that a few of the 3rd party libraries inside of it were already minified. Before minification the file was about 500kb in size.
[10:47] <evilstreak> the before minification size is pretty irrelevant
[10:47] <Dantman> ;) That's kindof why I have 3js files loading instead of one,
[10:47] <evilstreak> since it's affected by style and frequency of comments, amount of white space, choice of soft or hard tabs, etc
[10:48] <Dantman> Heh, ya I know.
[10:48] <Dantman> ^_^ But still.... over ten thousand lines of code... heh
[10:49] <Dantman> Of course... that doesn't compare to how inefficent the system itself was at rendering a page...
[10:49] <evilstreak> a lot of the code on this project is verbosely written (to a painful extreme). I spend most of my time deleting code rather than adding new stuff
[10:49] <Dantman> Eckhart's homepage could take 20s to render if it wasn't in memcached.
[10:50] <Dantman> ((That's why eckharttolle.com is using a static html export, not being hosted in kv2))
[10:51] <Dantman> evilstreak, Heh, maybe you're employer is lucky... That sounds similar to the kind of situation where rather than bothering to fix it, I start pushing for completely rewriting the whole thing...
[10:52] <Dantman> ;) Why do you think we have a v3 when v2 never made it to production...
[10:52] <Dantman> ((Other than the fact that we made some really bad design choices when we started building it))
[10:53] <evilstreak> we can't rewrite it, we'd have to explain to the client how we allowed it to get to this state
[10:53] <Dantman> heh
[10:53] <evilstreak> plus the offshore team would have a fit if we started changing stuff they've already implemented
[10:53] <evilstreak> (we can change the JS, since that's just drop-in)
[10:53] * Dantman is lucky that is is just about the sole developer of Kommonwealth.
[10:54] <Dantman> Sure, we have 3rd party libraries... But about 1% of v2 was written by another developer...
[10:55] <Dantman> As for v3 it'll probably only be the auth system (I'm thinking of contracting someone to help out with the integration with Facebook Connect, OpenID, and OAuth from other systems)
[10:56] <Dantman> :) Heh, and one of those 3rd party libraries was written by me.
[12:51] <hannesw> quick poll:
[12:52] <hannesw> what version of filesystem API do people/projects here implement?
[12:53] <ashb> fs-base
[12:53] <ashb> gpsee and hippo have that
[12:53] <ashb> and I think narwhall's is pretty similar
[12:54] <hannesw> is fs-base a strict subset of fs?
[12:54] <ashb> its meant to be i think, but things might be out of sync
[12:54] <ashb> I didn't go for fs because I dont agree with the number of methods it has
[12:54] <ashb> w.r.t path objects/strings
[12:55] <hannesw> ok
[12:55] <ashb> to me each new method is extra burden on implementors and users - code to maintain and docs to grok/write
[12:56] <hannesw> and fs-base lets you do everything you want to do?
[12:56] <ashb> yup
[12:56] <ashb> its not quite perfect when it comes to dealing with paths
[12:56] <hannesw> ok, i think we'll consider switching to fs-base, too
[12:56] <ashb> but its certainly doable
[12:56] <ashb> and the proposed fs module can be done in pure JS on top
[12:56] <ashb> or natively to make it faster
[12:57] <hannesw> ok, thanks!
[18:10] <oberhamsi> would someone show me a snippet that uses that client to simple GET http://example.com, please http://github.com/kriszyp/commonjs-utils/blob/master/lib/jsgi-client.js
[18:10] <oberhamsi> ..i'll learn about promises if i like what i see.. promised!
[18:16] <oberhamsi> ah found an example http://www.sitepen.com/blog/2010/03/02/commonjs-utilities/
[19:17] <Dantman> *sigh* Now I have to figure out why I'm having issues loading JMagick.
[19:43] <Dantman> Aaaagh... it won't load... Damnit runtime, libJMagick.so IS on the library path...
[19:48] * Dantman twitches at the idea of having someone write a java LiveConnect for SpiderMonkey...
[19:49] <Wes-> Dantman: What are you trying to do?
[19:50] <Dantman> Wes-, my actual issue, or my random tangent into the dark recesses of programmers insanity?
[19:52] <Dantman> ie: Was that in reply to my JMagick comments, or my SM LiveConnect
[19:52] <Wes-> both? :)
[19:52] <Dantman> Heh
[19:52] <Wes-> jmagick should be easy enough to debug
[19:52] <Wes-> ldd is your friend
[19:52] <Wes-> so is strace
[19:55] <Dantman> The later part was a random thought about how java and c connect through jni... iirc it should be possible for c programs to use jni to access java classes, thus if you're, erm... adventurous enough you should theoretically be able to replicate the Rhino LiveConnect API to provide JavaScript running inside of SpiderMonkey with the same access to java apis (sans-rhino classes)...
[19:56] <Dantman> Course that's not a good way for people to try to write File api in JS... But it is a good way to allow things written initially for Rhino to be run in SpiderMonkey before being ported.
[19:58] <Dantman> As for my real issue... I'm trying to get JMagick working in Rhino... But I'm getting a "UnsatisfiedLinkError: no JMagick in java.library.path" from the java classes trying to load the JNI despite including libJMagick.so inside the library path and confirming that it's there.
[19:58] <Wes-> that's completely insane
[19:58] <Wes-> but technically possible
[19:58] <Wes-> would require a JVM
[19:58] <Dantman> heh ;)
[19:59] <Wes-> Dantman: strace the JVM, see what it's trying to load
[19:59] <Wes-> (I treat the JVM just like any other C program for purposes of debugging)
[20:00] <MisterN> Wes-: you use gdb for that, too? :D
[20:01] <Wes-> MisterN: have been known to do that
[20:01] <MisterN> cool
[20:02] <MisterN> you really are crazy :)
[20:03] * Dantman struggles with the temptation to come up with some semi-philosophical quote about programmers, insanity, and how we make all the fancy stuff people use...
[20:07] <Dantman> Wes-, Do I read "read(3, "/usr/local/jmagick/lib/libJMagic"..., 128)" as read(handle, dataThatWasRead, lengthOfDataRead)?
[20:08] <Wes-> MisterN: Not crazy. The JVM is just a complicated C program with a very expressive configuration language
[20:08] <MisterN> Wes-: dunno why not use the java debugger?
[20:08] <Wes-> Dantman: 128 is probably the size of the buffer
[20:09] <Wes-> Dantman: return value will be amount of data read
[20:09] <Wes-> MisterN: I don't know java
[20:09] <Dantman> Ok
[20:12] * Dantman runs into issues most of the time he tries to install debuggers and tends to not use them as a result...
[20:45] <Dantman> Seeing other people with the same issue, but without a sollution isn't very nice either: http://dev.eclipse.org/newslists/news.eclipse.platform.swt/msg15332.html
[21:01] <Dantman> T_T This almost sounds like an issue to do with 64bit installations
[21:31] <Dantman> *sigh* I found away around the issue... I have to patch JMagick to make it use System.load() and an envvar instead of depending on System.loadLibrary()
[21:35] <MisterN> Dantman: still sure you want to use rhino? :P
[21:36] <Dantman> ^_^ It's worked fine for everything so far...
[21:36] <Dantman> s/everything/everything else/
[21:37] <Wes-> LD_PRELOAD=/path/library.so
[21:37] <Wes-> before launching the JVM
[21:41] <Dantman> Didn't work...
[21:41] <Dantman> This seams like a java thing...
[21:43] <Dantman> Problem is your only option for loading is System.loadLibrary... JMagick doesn't check if it's already loaded (and I doubt Java provides a good way to check) so it doesn't matter if you preloaded it.
[21:44] <Dantman> However, thanks for letting me know about that... If there was a way to make it so you could load it yourself instead that env var would work better than making code do a System.load call
[21:56] <Dantman> MisterN, I need File IO + Streams + seek + Binary + Worker Threads with a better queue system than what WebWorkers has + a Mongo Driver + md5 and sha512 + ImageMagick + Sockets + load() style importing (not require())
[21:56] <Dantman> What other option do I have?
[21:56] <MisterN> can all be done with flusspferd :)
[21:57] <Dantman> ;) And I accidentally hit enter while I was writing that list.
[21:57] <Dantman> MisterN, without hiring a C++ programmer?
[21:58] <MisterN> Dantman: you could _become_ a c++ programmer if you don't want to hire one :P
[21:59] <Dantman> And write something meant for production use as my first project?
[21:59] <Dantman> In a language distant from the programming conventions I'm used to...
[22:00] <MisterN> *shrug*
[22:01] <Dantman> Not that we won't hire a C/C++ programmer later...
[22:02] <MisterN> Dantman: sure, and in the meantime you have to fight with java. i guess that's your problem then.
[22:03] <Dantman> ^_^ I may have to fight with it... but at the same time it's also saved me a lot of time.
[22:03] <Dantman> Just, right now the target is actually getting Kommonwealth to an alpha/beta state. Getting into C++ and SpiderMonkey will take a huge chunk of time.
[22:07] <Dantman> The plan is to first build Kommonwealth on Rhino. Then work on MonkeyScript in Java getting it to the point it has what it needs to be able to run Kommonwealth. Then hire a C/C++ programmer to port the Java version of MonkeyScript.

 

 

Logs by date :