Mochabot log - CommonJS IRC channel: #commonjs on irc.freenode.net

2010-05-20:

[4:28] <Dantman|Away> Wes--, any tips for after you find out an old poorly configured server of yours has been sending out spam because one of the many php based systems you use is compromised?
[4:28] <Wes-> Dantman|Away: first things first, firewall off outbound port 25
[4:28] <Wes-> Dantman|Away: or rename the sendmail binary, assuming the mail is going out sendmail
[4:29] <Dantman|Away> :/ AWS doesn't make firewalling easy
[4:29] <Wes-> can you change the system's routing table?
[4:29] <Dantman|Away> iptables?
[4:29] <Wes-> iptables for firewalling
[4:29] <Wes-> route for routing
[4:31] <Wes-> probably something like iptables -I FORWARD 1 --dport 25 -j DROP, assuming packets from the local host traverse the FORWARD chain
[4:31] <Wes-> (I normally use iptables in an in-bewteen-appliance context)
[4:38] <Dantman|Away> Agh... I want to dispose of this server...
[4:38] <Dantman|Away> www-data 19148 0.0 0.1 4964 2156 ? S May11 0:00 perl /tmp/bp.pl 31337
[4:38] <Dantman|Away> www-data 19477 39.2 0.0 2776 1684 ? R May11 4984:33 ./exploit
[4:40] * Dantman|Away wonders if removing the sendmail package would be good enough
[4:47] <Dantman|Away> Aaagh... and now I get an error while trying to remove anything mail related
[8:08] <gmosx> good morning
[12:01] <ashb> Wes-: 'assuming packets from the local host traverse the FORWARD chain' they dont
[12:01] <ashb> you want the OUTPUT chain
[12:20] <Wes-> ashb: Ah! Right, I knew I was forgetting something
[14:03] <mschwartz> hmmm
[14:04] <mschwartz> ab -t 30 -k -c 10 http://localhost:8080/
[14:04] <mschwartz> Requests per second: 166.90 [#/sec] (mean)
[14:04] <mschwartz> that's for ringojs
[14:04] <mschwartz> ab -t 30 -k -c 10 http://localhost/
[14:04] <mschwartz> Requests per second: 3631.86 [#/sec] (mean)
[14:04] <mschwartz> that's for apache
[14:05] <mschwartz> ab -t 30 -k -c 10 http://localhost:8080/welcome/
[14:05] <mschwartz> Requests per second: 1347.76 [#/sec] (mean)
[14:05] <mschwartz> that's for helma 1.6.3
[14:06] * mschwartz is a huge fan of helma
[14:07] <jbrantly> what about node? :)
[14:10] <mschwartz> my benchmarks for node aren't so good
[14:10] <mschwartz> depends on what it's doing
[14:11] <mschwartz> I was looking at node for mongodb service
[14:11] <mschwartz> it was 3x slower than php+apache
[14:11] <mschwartz> however, I note that the apache benchmark I showed wasn't running any PHP, just serving static "It works!" page
[14:12] <mschwartz> in my testing of Helma, I've found I can make it do the same as PHP, but 10x faster if not more
[14:14] <mschwartz> I don't care for the baggage Helma brings, though
[14:14] <mschwartz> hop objects, etc.
[14:14] <mschwartz> the skins are ok
[14:16] <ondras> there is not much point in comparing js-based httpd and apache-based httpd
[14:16] <mschwartz> I'd love to see all that stripped out and commonjs support added (require, etc.)
[14:16] <mschwartz> there is a point
[14:16] <mschwartz> in my company, we do PHP
[14:16] <mschwartz> to advocate a switch, the performance is a key factor
[14:17] <ondras> in my opinion, any reasonably written js listening server will outperform apache
[14:17] <ondras> but that is not a reason to kill apache
[14:17] <ondras> apache offers other things, modules, years of experience and stability, ...
[14:18] <mschwartz> well, I agree to a large degree
[14:18] <ondras> to leave PHP, you do not need to leave apache, though
[14:18] <mschwartz> if you want to implement some feature that requires LDAP
[14:18] <mschwartz> you'll find it in PHP already
[14:18] <mschwartz> you have to implement it for Helma from scratch
[14:18] <mschwartz> or wrap Java classes
[14:19] <mschwartz> I imagine it would be much more work for a v8 solution
[14:26] <Wes--> mschwartz: I've been mixing and matching SSJS with PHP where it's convenient for me to do so. Biggest key for me was ability to read PHP session files.
[14:26] <mschwartz> haha
[14:26] <mschwartz> I avoid session like the plague
[14:26] <Wes--> out-and-out replacement is very expensive. Migration is cheap.
[14:26] <Wes--> mschwartz: Works for me. But my use is pretty light.
[14:27] <mschwartz> yeah, that's where we are at... as for session, it is a file I/O and serialization/deserialization every page fetch!
[14:27] <mschwartz> well, potentially serialization
[14:27] <mschwartz> and it fills up /tmp
[14:27] <Wes--> mschwartz: File I/O only if the implementation and your kernel sucks
[14:27] <mschwartz> Linux
[14:27] <Wes--> mschwartz: you should be able memory map it
[14:28] <Wes--> (I don't know about LInux, but I can't imagine it doesn't have a VM dirty page cache... solaris has had that for 10 years)
[14:28] <mschwartz> you can implement your own handler for sessions
[14:28] <mschwartz> but you loose the atomic nature of them
[14:28] <Wes--> *nod* - but if your session files are smaller than a memory page each, the default handler is probably about as fast as reasonable, or the implementation sucks. (I don't know which)
[14:28] <mschwartz> lose
[14:29] <mschwartz> my look at mongodb illustrates some of the issue
[14:29] <mschwartz> I can see queries take < 1 ms
[14:29] <Wes--> "but you lose the atomic nature of them" - describe?
[14:29] <mschwartz> but turning json strings into objects takes 5ms
[14:29] <mschwartz> atomic nature
[14:30] <mschwartz> if you use the built in sessions
[14:30] <mschwartz> if you have two simultaneous accesses, as in ajax, one will block
[14:30] <mschwartz> if you don't, both will run and you can trash the session
[14:30] <Wes--> Right, this is because PHP uses fcntl advisory locking to hold the session file
[14:31] <mschwartz> exactly
[14:31] <mschwartz> so if you implement MySQL backed sessions
[14:31] <mschwartz> you need to do a select for update
[14:31] <mschwartz> to achieve a lock
[14:31] <Wes--> which is kind of sucky, actually - that's caused us problems in the past. My JS code knows how to cheat there (we NEVER write, only read)
[14:31] <mschwartz> the downside of PHP, IMO, is it's not an application server
[14:32] <mschwartz> with something like helma
[14:32] <mschwartz> you can store session in RAM
[14:32] <Wes--> Assuming MySQL is ~SQL 99, you can also use transactions rather than select for update, which /might/ be cheaper (you'd need to measure)
[14:32] <Wes--> And you're right about PHP - it's not an application server
[14:32] <Wes--> It's basically all the kitchen sinks they could find in the neighbourhood, smashed together and formed into a bathtub
[14:32] <mschwartz> you can use apc ram cache
[14:33] <mschwartz> or something like it
[14:33] <mschwartz> but I think you are doing serialize/deserialize under the hood
[14:33] <mschwartz> so my boss tells me, "I want mongo!"
[14:33] <mschwartz> and "I want nodejs"
[14:34] <Wes--> If you don't serialize/deserialize under the hood, you have extremely complicated problems trying to keep PHP objects alive across processes and/or apache pools
[14:34] <mschwartz> well, you can't do true application server things with apc cache
[14:34] <mschwartz> like
[14:34] <mschwartz> you can't keep an open socket
[14:34] <Wes--> mschwartz: Nothing wrong with that, assuming he has a reason. Personally, I dislike being handed solutions, I prefer to be presented with problems. :)
[14:34] <mschwartz> ha
[14:35] <mschwartz> so I spent 2 days playing with nodejs
[14:35] <mschwartz> I have a few serious criticisms of it
[14:35] <mschwartz> I have quad core
[14:35] <mschwartz> it doesn't use moret han one
[14:35] <mschwartz> I get white screens a LOT
[14:35] <Wes--> That doesn't surprise me
[14:35] <mschwartz> the eclipse debugger causes it to core dump
[14:35] <Wes--> It's basically co-routines, right - are you familar with GNU pth? Same problems
[14:35] <mschwartz> yes
[14:35] <mschwartz> core dump is a HUGE issue in a web app
[14:36] <Wes--> As for white screens/eclipse - I'm familiar with node on an API and technology POV; haven't actually used it
[14:36] <Wes--> no kidding
[14:36] <mschwartz> if you have a reference error
[14:36] <mschwartz> you get white screen
[14:36] <mschwartz> I suppose you can implement a try catch or exception handler
[14:36] <mschwartz> and try to render a useful trace and error message
[14:36] <Wes--> Do you know if ryan is planning support for worker threads? That could help with the multi-core use
[14:37] <mschwartz> the nodejs.org site talks about such plans
[14:37] <Wes--> mschwartz: If it's dumping core, I don't see how you could reasonably catch that
[14:37] * Wes-- nods
[14:37] <mschwartz> you can't catch it
[14:37] <mschwartz> but you can install a cron task to restart it if it stops.... LOL
[14:37] <Wes--> I'm in the middle of a big GPSEE refactor to support worker threads. They add an interesting dimension to the common js module ecosystem
[14:37] <Wes--> mschwartz: if you're stuck at that level, don't use cron: start it from /etc/inittab
[14:38] <Wes--> (inittab in respawn mode, make sure node doesn't background itself)
[14:38] <mschwartz> heh
[14:38] <mschwartz> it really shouldn't core dump
[14:38] <Wes--> Well, no, core dumps are always bugs
[14:38] <Wes--> Nothing should have bugs
[14:38] <mschwartz> but that speaks to production ready
[14:38] <Wes--> Fact is, things DO
[14:38] <Wes--> It does - CommonJS is /young/
[14:39] <mschwartz> an observation about commonjs
[14:39] <Wes--> I mean, I can remember when Apache's predecessor was unstable like that too
[14:39] <Wes--> But, I mean, that was like in 1996
[14:39] <mschwartz> include() should be in the spec
[14:39] <mschwartz> it's like require, but runs in the caller's address space
[14:39] <Wes--> mschwartz: plenty of differing opinion on that one - I'd direct you to read the newsgroup archives
[14:39] <mschwartz> oh, I read it
[14:39] <mschwartz> I don't want to put:
[14:39] <mschwartz> var fs = require('fs');
[14:40] <mschwartz> var db = require('db');
[14:40] <mschwartz> in EVERY js file
[14:40] <mschwartz> I want to include('my_requires');
[14:40] <mschwartz> which does all that
[14:40] <mschwartz> so I can change what gets included in one place
[14:40] <Wes--> require("myrequires").init(this)
[14:40] <mschwartz> maybe it can be abused
[14:41] <Wes--> something like that should get you most of the way there
[14:41] <mschwartz> so I mentioned the helma baggage
[14:42] <Wes--> Which one is helma again? The mature product built around Rhino?
[14:42] <mschwartz> it has app.addRepository()
[14:42] <mschwartz> yes
[14:42] <mschwartz> but it only has one static directory per application
[14:42] <mschwartz> app.addRepository() is roughly "include"
[14:43] <mschwartz> if it looked more like a traditional www server
[14:43] <mschwartz> I think it'd be quickly adopted
[14:44] <Wes--> *nod* - little secret, I am still using CGI to call my CommonJS web stuff (our main focus isn't web). It's very comfortable.
[14:44] <mschwartz> my ideal server would probably be rhino based
[14:44] <mschwartz> rhino?
[14:44] <Wes--> My long-term plans are probably to build an apache module which exposes apache internals to JS
[14:44] <mschwartz> it would serve static files
[14:44] <Wes--> then build something in JS which can work as a content handler
[14:44] <Wes--> No, I use spidermonkey
[14:44] <mschwartz> if the file requested is .sjs (server-side js), it executes it
[14:45] <oberhamsi> mschwartz, helma devs are now working on ringojs
[14:45] <Wes--> Sounds about what I want as well
[14:45] <oberhamsi> didn't follow your discussion.. what you wanna do.. reading now
[14:45] <mschwartz> ober: see my benchmarks above?
[14:45] <mschwartz> if .sjs, the script can include() or require() what it wants
[14:45] <Wes--> Although, like I said -- web isn't our priority, and I want to coexist in my existing apache/php/C/shell/perl ecosystem
[14:46] <Wes--> (thank GOD I got the tomcat out of there)
[14:46] <mschwartz> heh
[14:46] <oberhamsi> mschwartz, ah yes you are looking for speed. are you running ringojs git or release?
[14:46] <mschwartz> release
[14:46] <dmachi> mschwartz: aside from the require()/include() thing you mention, narwhal is pretty straight forward to setup/configure and is similar in configuration to a lot of other things like django/rails/etc. Also many of the package/require specifications are debated heavily and the goal is to not eliminate the browser as well.
[14:46] <mschwartz> the .tar.gz file, 0.5
[14:46] <Wes--> tomcat was so slow I had to use apache to serve static content up and proxy through to tomcat
[14:46] <oberhamsi> oh okay
[14:46] <Wes--> I mean, that's just ridiculous
[14:47] <mschwartz> one thing I like about helma is
[14:47] <mschwartz> unzip ; cd ; ./start.sh
[14:47] <dmachi> Wes--: hehe
[14:47] <mschwartz> it runs on windows or unixen
[14:47] <Wes--> That IS nice. Even better than apxs :)
[14:48] <oberhamsi> mschwartz, yes that bench looks terrible. can't say much about it, hannesw in #ringojs might help
[14:48] <dmachi> persvr is like that too
[14:48] <oberhamsi> helma is super stable, ringojs is 0.5 ... :)
[14:48] <mschwartz> oh, I know ringojs is new
[14:48] <oberhamsi> care to show your test ?
[14:48] <mschwartz> I am suggesting here that focus on Helma 1.7.x
[14:48] <Wes--> mschwartz: I also would bet that you could get some better perf out of helma with a different JVM. But I am no expert in such matters..
[14:48] <mschwartz> upgrade it
[14:48] <mschwartz> ober:
[14:48] <mschwartz> the benchmark is:
[14:49] <mschwartz> ringojs demo app home page
[14:49] <mschwartz> apache home page (no php, just "it works!" page)
[14:49] <mschwartz> helma welcome app
[14:49] <mschwartz> ab with same parameters
[14:49] <mschwartz> all done localhost on the same system
[14:51] <oberhamsi> mschwartz, okay .. i'm checking if anything terribly slow happens in demo app
[14:51] <mschwartz> seems like demo app shouldn't be slow
[14:52] <mschwartz> it's rather basic WWW pages
[14:52] <oberhamsi> demo app is nested app.. some of the lower apps like jsdoc are very expensive
[14:53] <mschwartz> I'm thinking maybe this could be built into helma:
[14:53] <mschwartz> http://hsqldb.org/
[14:54] <mschwartz> something tricky in rhino/helma is converting an object to map to pass to a Java method
[14:54] <mschwartz> seems like it should be automagic
[14:54] <mschwartz> hey hannes
[14:57] <mschwartz> (8:01:12 AM) mschwartz: I'm thinking maybe this could be built into helma:
[14:57] <mschwartz> (8:01:12 AM) mschwartz: http://hsqldb.org/
[15:06] <Wes--> mschwartz: Looking at your benchmark: why -c 10? Why not -c 100?
[15:06] <mschwartz> -c 100 would work, too
[15:07] <mschwartz> but -c 10 is a reasonable number for the first test
[15:07] <Wes--> I would expect it to be a more realistic bench for a loaded system, FWIW - but that is just a gut feeling, I don't have any measurements to back it up
[15:07] <Wes--> Big question as to how it will affect the benchmarks, though - where bottle neck is
[15:08] <mschwartz> I'd think -c 10 is a good test for the server side execution time
[15:08] <mschwartz> you're fetching just the generated HTML
[15:08] <mschwartz> -c 100 might be a good test to see what happens with all the static fetches
[15:08] <mschwartz> sorta
[15:09] <Wes--> FWIW, -c10 vs. -c100 over here makes a ~12% difference. Although, my test is biased toward that -- spawn a CGI, start javascript, print the current time, exit
[15:09] <mschwartz> 12% faster?
[15:09] <Wes--> yeah
[15:09] <mschwartz> trying -c 100 on helma
[15:10] <mschwartz> -c 10:
[15:10] <mschwartz> Requests per second: 1347.76 [#/sec] (mean)
[15:10] <mschwartz> -c 100:
[15:10] <mschwartz> Requests per second: 984.02 [#/sec] (mean)
[15:10] <Wes--> Is that serving static content or generated content?
[15:11] <mschwartz> generated
[15:11] <mschwartz> the helma welcome app
[15:11] <mschwartz> at -c 100, it may be a server tuning issue
[15:11] <mschwartz> like max child threads allowed
[15:11] <Wes--> Interesting. That may mean it's somewhat concurrency-bound, but yeah, server tuning is a big issue there
[15:12] <Wes--> The server tuning for my tests today consistent of "apt-get install apache2" :D
[15:12] <Wes--> s/consistent/consisted
[15:13] <Wes--> (anybody know how to find out what kind of CPU / how many cores on a linux box?)
[15:14] <mschwartz> cat /proc/cpuinfo
[15:14] <Wes--> mschwartz: thanks, I keep forgetting about that
[15:14] <Wes--> well, about the whole procfs on linux
[15:14] <Wes--> I must have a memory defect there :)
[15:17] <hannesw> mschwartz you're interested in helma?
[15:17] <mschwartz> hannes: absolutely
[15:17] <mschwartz> I've been using it since 1.6.0
[15:17] <mschwartz> I am a HUGE fan
[15:17] <hannesw> not that i want you talk out of it, but ringojs is where new things are happening
[15:17] <hannesw> well nice to hear :)
[15:18] <mschwartz> let me paste in my benchmarks from earlier
[15:18] <mschwartz> 7:12:00 AM) mschwartz: ab -t 30 -k -c 10 http://localhost:8080/
[15:18] <mschwartz> (7:12:18 AM) mschwartz: Requests per second: 166.90 [#/sec] (mean)
[15:18] <mschwartz> (7:12:23 AM) mschwartz: that's for ringojs
[15:18] <mschwartz> (7:12:37 AM) mschwartz: ab -t 30 -k -c 10 http://localhost/
[15:18] <mschwartz> (7:12:46 AM) mschwartz: Requests per second: 3631.86 [#/sec] (mean)
[15:18] <mschwartz> (7:12:49 AM) mschwartz: that's for apache
[15:18] <mschwartz> (7:13:01 AM) mschwartz: ab -t 30 -k -c 10 http://localhost:8080/welcome/
[15:18] <mschwartz> (7:13:13 AM) mschwartz: Requests per second: 1347.76 [#/sec] (mean)
[15:18] <mschwartz> (7:13:16 AM) mschwartz: that's for helma 1.6.3
[15:19] <mschwartz> looks like helma 1.6.3 is 10x faster...
[15:19] <mschwartz> (than ringojs)
[15:19] <hannesw> well, you have to look at what the app does, you know
[15:19] <mschwartz> it's the welcome app's home page
[15:19] <mschwartz> and the demo app's home page
[15:20] <mschwartz> pretty plain looking HTML pages
[15:20] <hannesw> helma welcome app is basically a static page, ringojs demo page does a lot (template parsing, markdown, gzip encoding,...)
[15:20] <mschwartz> so I was talking about a few suggestions for helma
[15:21] <mschwartz> I wrote a CMS demo using helma 1.6.3
[15:21] <mschwartz> I generate pages in .006 seconds
[15:21] <mschwartz> that's doing 5 template parsing
[15:22] <mschwartz> several SQL queries (8?)
[15:22] <mschwartz> not using skins
[15:22] <Wes--> helma and ringo will serve up the same content, right? Can you bench with the same content rather than different pages?
[15:22] <mschwartz> using a lightweight javascript template engine
[15:23] <mschwartz> in fact
[15:23] <mschwartz> I'll do the ab on my CMS
[15:24] <hannesw> mschwartz: there's also a #helma channel on freenode
[15:25] <hannesw> helma is not really commonjs
[15:25] <mschwartz> I get what Helma is
[15:25] <mschwartz> it's a hop object server
[15:25] <mschwartz> you want me to talk about my ideas in #helma?
[15:26] <hannesw> yes, i think that would be the better place
[15:26] <mschwartz> ok
[16:40] <kuya> OT, does anyone know of a js dom implementation? iirc there was one announced semi recently
[16:41] <kuya> ahah http://github.com/tmpvar/jsdom
[16:41] <kuya> and it was on topic! :)
[16:41] <Wes--> envjs count?
[16:42] <kuya> oh nice
[16:42] <kuya> thanks Wes--
[16:42] <kuya> now i have a choice...
[16:43] <Wes--> there's at least one more out there
[16:43] * Wes-- thinks
[16:43] <Wes--> by nickgsuperstar maybe?
[16:44] * Wes-- can't seem to find it
[16:44] <kuya> its ok - 2 is enuff for now :)
[17:55] <kuya> *grumble*
[17:55] <kuya> cant find much usefull on js spiders
[18:03] <Wes--> well, they all have 8 legs
[18:03] <Wes--> if you want to know more, consult wikipedia
[18:04] <kuya> :)

 

 

Logs by date :