Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2010-02-08:

[0:26] <ryah> ashb: donno
[20:16] <Wes--> observation: it is impossible for a common js module to serialize a class provided by a non-top-level module
[20:17] <Wes--> ({ new (require("./myFancyClass"))(details) })
[20:17] <Wes--> doesn't work, due to ./
[20:17] <Wes--> ideas?
[20:18] <Wes--> I guess this means that we need to define the result of module.id as being always capable of reloading that same module?
[20:22] <Dantman> Serialize... People actually use that moz extension for non-debugging purposes in real code?
[20:24] <Wes--> dantman: What moz extension? I thought JSON object was part of ES5?
[20:24] <Dantman> I'm talking about toSource/uneval
[20:24] <Dantman> Since when to we serialize code in JSON?
[20:25] <Wes--> I'm talking about serializing objects
[20:28] <Dantman> JSON has always been independent of classes; Either the object gets turned into a plain object literal with it's enumerable properties stringified. Or it implements a custom toJSON which outputs a custom literal object, string, etc... There is never any implied reverse.
[20:29] <Wes--> Well, that's just stupid
[20:29] <Dantman> If you have a class you intend to pass as JSON back and forth. You implement a toJSON on it. And when you get JSON data back, you JSON.parse it and create the object yourself.
[20:29] <Wes--> Okay, then, I invent my own serialization representation, voila
[20:30] <Wes--> The problem with module IDs still exists
[20:34] <ondras> Wes--: you can use the module.uri property
[20:34] <ondras> as long as your require() implementation supports absolute paths :)
[20:34] <ondras> (as v8cgi does)
[20:34] <Wes--> Right, but that's not a /requirement/ yet, is it?
[20:34] <ondras> no
[20:34] <ondras> it is some kind of extension or what
[20:34] <Wes--> I'm thinking if we make require(module.id) "must work" then we solve this problem completely
[20:35] * ondras is not a big fan of this way of serialization, though
[20:36] <ondras> the idea of (new ByteArray).toSource() == "new require('binary').ByteArray()" is something I do not welcome with joy
[20:36] <Wes--> ondras: why not?
[20:36] <ondras> because it makes the whole serialization dependant on stuff which I believe is not related
[20:37] <ondras> what if this string is to be passed to some environment where there is no require?
[20:37] <Wes--> ....how would YOU toSource a ByteArray?
[20:37] <ondras> nowai
[20:37] <Wes--> ondras: If there is no require, there is no ByteArray
[20:37] <ondras> no toSource for me, thanks
[20:37] <ondras> the toJSON is a fine approach, if you ask me
[20:37] <Wes--> JSON cannot create ByteArrays
[20:38] <ondras> but bytearrays know the best how to jsonify themselves
[20:38] <Wes--> That doesn't make sense
[20:38] <Wes--> Let's say I have a tree of objects, and I want to store it on disk
[20:38] <ondras> btw, I finished reading The Good Parts recently
[20:38] <ondras> thanks for the idea for that book :)
[20:39] <Wes--> file.write(object.toSource()) completely stores the state if every object knows how to toSource() itself properly
[20:39] <Wes--> ondras: You're welcome! It's a great book.
[20:39] <ondras> Wes--: funny - I am just implementing something similar to my roguelike save/load capability
[20:39] <Wes--> ondras: I use that approach regularly to snapshot application state
[20:40] <Wes--> Having pervasive toSource() is *so helpful* in that regard
[20:41] <Wes--> All I have to do is make sure that application state does not depend on values of local variables
[20:41] <ondras> (and attached events :) )
[20:41] <Wes--> And suddenly I can migrate running apps
[20:41] <Wes--> Yeah, and a trillion other things, not to mention the program counter
[20:41] <ondras> the thing here is that you have to execute your constructors in deserialization phase
[20:41] <Wes--> But it's not so bad for OLTP type systems
[20:41] <ondras> which might not be wanted
[20:42] <ondras> to instead, I prefer when everyone can .toJSONify itself
[20:42] <ondras> storing its state, not a recipe for revival
[20:42] <Wes--> But if there is no recipe for revival... how do you revive?
[20:42] <ondras> well, I know a constructor and a complete internal state
[20:43] <ondras> using something like "begetObject", I create an instance withouth calling the constructor
[20:43] <Wes--> But then you need to store type information along with the data
[20:43] <ondras> and restore its state
[20:43] <ondras> Wes--: yeah.
[20:43] <ondras> but you do the same
[20:43] <ondras> plus
[20:43] <Wes--> beget vs new -- 6 of one, half a dozen of the other
[20:43] <ondras> you need to execute the ctors :)
[20:44] <Wes--> I personally don't see much of a difference between X.beget() and new X()
[20:44] <Wes--> they are both recipes for creating objects from templates
[20:44] <ondras> when your constructing function does not accept arguments
[20:44] <ondras> nevertheless
[20:44] <ondras> in my game, there are relatively lots of classes
[20:44] <ondras> (200+)
[20:44] <ondras> and it is bad for maintenance to write .toSource() for each one
[20:44] <Wes--> How do you plan to serialize them?
[20:44] <ondras> so I just jsonify them implicitely
[20:45] <ondras> it is ~200kB game data
[20:45] <ondras> :)
[20:45] <ondras> (that's why I implemented a LZW compressor)
[20:45] <Wes--> By iterating over the own props?
[20:45] <ondras> yeah
[20:45] <ondras> this is my second take on this
[20:46] <ondras> first, I created some serialization routine where necessary
[20:46] <ondras> and I ended with ~30-40 serialization routines
[20:46] <Wes--> How do you get the right methods attached to the de-serialized objects?
[20:46] <ondras> so I decided that was unmaintainable
[20:46] <ondras> Wes--: I store separately a list of constructors and each deserialized object contains index reference to its constructor
[20:46] <Wes--> And, BTW, "iterate over own props" from either toJSON or toSource is an equivalent problem IMO
[20:47] <ondras> so I know what to beget
[20:47] <Wes--> Ah, that makes sense
[20:47] <ondras> Wes--: well, that is true, the difference is imho when you restore
[20:47] <Wes--> You just don't want to be able to eval() the final result
[20:47] <ondras> it is difficult
[20:47] <ondras> many circular references there
[20:48] <Wes--> Also I suspect you're not dealing with trees of objects
[20:48] <ondras> exactly.
[20:48] <ondras> if you want, I can show you the game data
[20:48] <ondras> both uncompressed and compressed
[20:48] <ondras> it is not a nice look :))
[20:48] <Wes--> I'm sure, lol
[20:49] <ondras> http://bespin.cz/~ondras/js-like/
[20:49] <ondras> pick a char
[20:49] <ondras> and when game starts
[20:49] <ondras> type test2() into console
[20:49] <Wes--> Just trying to understand why you're so again something as elegant as newObjectGraph = eval(oldObjectGraph.toSource())
[20:49] <ondras> or just try the 'S'ave button to see compressed data saved into localStorage
[20:50] <ondras> Wes--: circular references?
[20:50] <ondras> I know there is some mozilla JSON extension for these
[20:50] <ondras> but...
[20:50] <Wes--> ondras: I don't think there is a problem with circular references in toSource.. lemme test
[20:51] <Wes--> ondras: Ah, right, sharp objects
[20:51] <Wes--> v8 doesn't support?
[20:51] <ondras> Wes--: also, your toSource / constructor must be able to handle all internal variables
[20:52] <ondras> Wes--: not sure, this is not aimed at v8
[20:52] <ondras> iirc, all current browser run my game
[20:52] <ondras> Wes--: how do you toSource when your ctor has 2 arguments but 10 another properties are maintained via setters/getters?
[20:52] <Wes--> ondras: http://pastebin.mozilla.org/701752 fwiw
[20:53] <Wes--> ondras: You re-design the object
[20:53] <ondras> yeah
[20:53] <ondras> (yeah as per the pastebin)
[20:53] <ondras> "re-design the object" <-- ?
[20:53] <Wes--> Easiest re-design is to allow a complete state-restore with either the constructor a beget-like static method of the constructor function
[20:54] <ondras> hmm
[20:54] <ondras> obj.prototype.toSource = function() { return "obj().createFromData(" + this.a +", " + this.b + ", " + this.c + ")"; }
[20:54] <ondras> s/()//
[20:54] <Wes--> Right, that would work
[20:55] <ondras> that is an awful amount of maintenance imho
[20:55] <ondras> add one property
[20:55] <ondras> + add it to toSource
[20:55] <ondras> + add it to createFromData
[20:55] <ondras> ...
[20:55] <Wes--> ondras: just iterate over your properties
[20:56] <ondras> fine, we are slowly getting to what I do :)
[20:56] <ondras> so the difference is in the way construtor functions are stored
[20:56] <ondras> *constructor
[20:56] <Wes--> And that you are not able to store arbitrary object graphs AFAICT
[20:57] <ondras> only those containing references to instances from a well-known class list
[20:57] <ondras> and, of course, callback properties are pain
[20:57] <ondras> and attached events
[20:57] <ondras> .. :)
[20:57] <Wes--> But you can't, say, serialize a plain object containing arrays of your own objects without writing a bunch of extra code
[20:58] <Wes--> And things like attached events -- forget it, that's the wrong problem space entirely
[20:58] <ondras> there is always some place where extra code - relevant to serialization - will be placed
[20:58] <ondras> I decided to have it in 1 place
[20:58] <ondras> in some dedicated serializer
[20:58] <ondras> which iterates etc
[20:58] <ondras> resolves circulars
[20:58] <ondras> ...
[20:59] <ondras> the other option is that this serialization-relevant code is defined with each object
[20:59] <ondras> I am still not entirely happy with either solution
[20:59] <Wes--> *nod* - that's a design philosophy question largely
[20:59] <ondras> but after implementing both of these, the current one seems somewhat nicer to me
[20:59] <Wes--> I have been a fan lately of "objects know everything about themselves"
[20:59] <Wes--> (including how to clone themselves)
[20:59] <ondras> yes
[21:00] <ondras> that looks like a more systematic approach
[21:00] <ondras> ...until there are many of such objects.
[21:00] <Wes--> A big reason for me to like the small-island approach is that I have spent many years maintaining monoliths
[21:00] <Wes--> monoliths where only ~10% of the old code is still "active"
[21:01] <Wes--> but we have to support 100% of the datastructures et al because of the monolithic architecture
[21:01] <ondras> btw, I guess you are not aware of a FAST Burrows-Wheeler (javascript) inverse transformator?
[21:01] <Wes--> no
[21:01] <ondras> okay
[21:01] <ondras> there is very little done in compression are in JS :)
[21:01] <ondras> *area
[21:02] <Wes--> ondras: why do you need one?
[21:02] <ondras> I would like to improve my LZW
[21:03] <ondras> my JSON is about 1/5 of its original size now
[21:03] <Wes--> ondras: Have benchmarked real-world-data results against using Content-Transfer-Encoding: gzip ?
[21:03] <ondras> but ZIP is still FAR better
[21:03] <ondras> Wes--: I need (want) to compress data before sending it to server
[21:04] <Wes--> ondras: I would hazard a guess that decreasing the payload size with gzip and using native-code decompression will perform better than scripted bzip2 for a singificant proportion of real-world test cases
[21:04] <Wes--> OH
[21:04] <Wes--> Damn, I thought you were going the other way
[21:04] <Wes--> *hmm*
[21:04] <ondras> say your savegame has ~600kB of JSON
[21:04] <ondras> that is very unpleasant
[21:04] <ondras> here in .cz, most people use adsl...
[21:04] <ondras> that is about .5mbit upstram
[21:05] <Wes--> Yes, so you're talking about a 10 second upload
[21:05] <ondras> exactly
[21:05] <Wes--> can you post a link to a one of your save game files?
[21:05] <ondras> Wes--: not yet, you have to create one
[21:05] <ondras> Wes--: just hit the 'S'ave button
[21:06] <ondras> if you use the localStorage, you will see plain compressed data
[21:06] <ondras> if you use the clipboard option, they will get base64 encoded
[21:06] <Wes--> ondras: SONifying...
[21:06] <Wes--> Compressing...
[21:06] <Wes--> FAILURE: Error: Code @ 17117 > 255
[21:06] <ondras> baaad wes
[21:06] <ondras> using come codepoint > 255 :)
[21:07] <Wes--> ondras: I never even touched the keyboard
[21:07] <Wes--> baaaad ondras
[21:07] <ondras> oops?
[21:07] <Wes--> not testing on firefox
[21:07] <ondras> Wes--: http://bespin.cz/~ondras/data
[21:07] <ondras> here you are
[21:08] <Wes--> ondras: are you able to paste a decompressed version of that?
[21:08] <ondras> Wes--: http://bespin.cz/~ondras/data2
[21:10] <Wes--> ondras: Observation, you can cut down probably 20% on that data *pre* compression with some changes to your storage format
[21:10] <Wes--> - strict JSON says quotes are necessary, but not true if you use eval() and don't have wierd prop names
[21:10] <ondras> yeah
[21:10] <Wes--> - var N=null, then use N instead of null
[21:10] <ondras> I know, this is the first working implementation
[21:10] <ondras> no optimizations so fra
[21:10] <ondras> *far
[21:11] <Wes--> observation: you are already using sharp objects to resolve circular references
[21:11] <ondras> kind of :)
[21:11] <ondras> note those are strings
[21:11] <Wes--> oh yeah
[21:12] <Wes--> observation: you have a lot of repetition in prop names, a clever lookup table could probably save you a pile of over-the-wire data too
[21:12] <ondras> (that is what dictionary compression was designed for :) )
[21:13] <ondras> but yes, point taken
[21:13] <Wes--> Your encoding format might also be shrinkable by increasing the character set, looks like you are doing ~ base 64?
[21:13] <ondras> by default, the LZW output are bytes
[21:14] <Wes--> http://bespin.cz/~ondras/data isn't bytes thought
[21:14] <ondras> no
[21:14] <Wes--> base64(bytes) ?
[21:14] <ondras> exactly.
[21:14] <ondras> but that is only for clipboard storage
[21:15] <ondras> Wes--: btw, what browser have you tried? I am puzzled by that high codepoint error...
[21:15] <Wes--> base64 is about 6/8 effecient, you can do better.... was thinking a iso-8859-1 -> utf-8 transcode would be very close to 100% effecient but a few characters (namely 0) become a problem going into the clipboard
[21:15] <Wes--> ondras: Firefox 3.5
[21:15] <ondras> wow
[21:15] * ondras the same
[21:16] <ondras> and still, no issues
[21:16] <ondras> well, it is not relevant now, I will sanitize that later
[21:16] <Wes--> ondras: ugh, you have a heisenbug
[21:16] <ondras> Wes--: the size of base64 does not matter that much, because in this scenario, user just copies the data to a local file via clipboard :)
[21:16] <Wes--> ondras: worked this time, both times I just clicked on a character and then clicked on save
[21:17] <Wes--> ondras: true that!
[21:17] <ondras> I initially thought that I can put arbitrary data to a textarea
[21:17] <ondras> including String.fromCharCode(0)
[21:17] <ondras> but I was mistaken :)
[21:18] <ondras> or at least the clipboard on my work kubuntu was not happy with those
[21:18] <Wes--> Yeah, funny how that happens, eh? :)
[21:18] <ondras> :}
[21:20] <ondras> Wes--: so, thanks for consultations and considerations, it was a nice chat
[21:20] <ondras> time to go to bed now
[21:20] <Wes--> ondras: My pleasure! It's nice for me to change mental gears mid-day. :)
[21:20] <ondras> more compression benchmarks awaiting tomorrow :)
[21:21] <ondras> dtach

 

 

Logs by date :