LSD::RELOAD

July 9, 2008

my day job now involves world of warcraft

Filed under: Uncategorized — Leo Simons @ 6:43

Joost (where I work) is working with SK Gaming on doing live coverage of world of warcraft raids (announcement).

I haven’t quite convinced anyone yet that this means that actually playing world of warcraft should be part of my job description, but I do get to watch SK in-game footage as part of testing in-development versions of our software!

Working a bit on this stuff really makes me want to get back into raiding. If only I could find the time! In the meantime this is probably the closest I’ll get…

So, in the meantime I’m hoping we can get actual live coverage of some powerleveling (which I once tried to get good at). Apparently some people can do 1-70 in two days now! S(l)ick.

May 1, 2008

Serious PHP, part 1

Filed under: Uncategorized — Leo Simons @ 22:33

I used PHP 3 a lot back in the day, and I built some seriously interesting stuff with PHP 4 before moving on to what was then consider more “serious” technology. Recently I’ve been looking at PHP again, just to figure out how the famous LAMP stack works these days. Care to follow along?

In the below I assume you’re on Mac OS X and you sort-of know your way around unix.

Installing a bunch of packages under Mac OS X

We’re going to use a whole bunch of stuff…let’s get the package in one go (hint: you can install multiple ports at a time, see man port)

  • Install MacPorts
  • Install apache2:
    sudo port install apache2
  • Install MySQL 5:
    sudo port install mysql5
    (You will need to follow some on-screen instructions to finish configuration)
  • Install SQLite:
    sudo port install sqlite3
  • Install Perl 5.8:
    sudo port install perl5.8
  • Install Apache-Test:
    sudo port install p5-apache-test
  • Install PHP 5:
    sudo port install php5 +apache2+mysql5+pear+sqlite
    (you may want to add some more options, see what’s available using port variants php5; You may also need to follow some on-screen instructions to enable the extension within apache)
    sudo cp /opt/local/etc/php.ini-recommended /opt/local/etc/php.ini
  • Configure pear:
    sudo pear config-set php_ini /opt/local/etc/php.ini
    sudo pear upgrade-all
    sudo pear channel-update pear.php.net
    sudo pear channel-update pecl.php.net
  • Install APC:
    sudo pear install APC
  • Install Memcached:
    sudo port install memcached
  • Install memcache for python:
    sudo port install php5-memcache
    (You will need to follow some on-screen instructions to enable the extension)

Setting up MySQL replication

MySQL replication is a technique that helps with scaling up your application. Sooner or later your database tends to become a bottleneck, and in that case what you do is make sure your SELECT statements go to database slaves. It is easier to get this right from the start, rather than re-write your application later. So, set up some replication locally…Buy and read High Performance MySQL to get a good grasp for the details; it’s the best book on the subject.

After you’ve RTFMed a bunch, assuming you have a basic mysql install set up…here’s the magic…

# set the root password
mysqladmin -u root password 'blaat'

# shortcut
my="mysql -uroot -pblaat"

# db we'll be using
echo "DROP DATABASE IF EXISTS test;" | $my
echo "CREATE DATABASE test;" | $my
echo "GRANT ALL
  ON test.* TO 'test'@'%'
  IDENTIFIED BY 'test'" | $my

# prep for replication
echo "GRANT
    REPLICATION SLAVE,
    REPLICATION CLIENT
  ON *.* TO 'repl'@'%'
  IDENTIFIED BY 'blaat'" | $my

# I always manage to forget this one...
echo "FLUSH PRIVILEGES;" | $my

# enable binary logging and set server-id on to-be-master
sudo cat > /opt/local/etc/mysql5/my.cnf <<END
[mysqld]
log-bin
server-id = 1
END

# create config for first slave to run on port 3307
sudo cat > /opt/local/etc/mysql5/my_3307.cnf <<END
[mysqld]
server-id = 2
master-host = 127.0.0.1
master-user = repl
master-password = blaat
master-port = 3306
port = 3307
socket = /opt/local/var/run/mysql5/mysqld_3307.sock
pid-file = /opt/local/var/db/mysql5_3307/my.pid
datadir = /opt/local/var/db/mysql5_3307
log-error = /opt/local/var/db/mysql5_3307/my.error.log
END

# create config for second slave to run on port 3308
sudo cat > /opt/local/etc/mysql5/my_3307.cnf <<END
[mysqld]
server-id = 2
master-host = 127.0.0.1
master-user = repl
master-password = blaat
master-port = 3306
port = 3308
socket = /opt/local/var/run/mysql5/mysqld_3308.sock
pid-file = /opt/local/var/db/mysql5_3308/my.pid
datadir = /opt/local/var/db/mysql5_3308
log-error = /opt/local/var/db/mysql5_3308/my.error.log
END

# stop the current mysql database
sudo killall mysqld_safe

# remove any previous/old binary log files
rm /opt/local/var/db/mysql5/*-bin*

# initialize data directories for the slaves
mkdir /opt/local/var/db/mysql5_3307
chown mysql:mysql /opt/local/var/db/mysql5_3307
rsync -av /opt/local/var/db/mysql5/ /opt/local/var/db/mysql5_3307/
mkdir /opt/local/var/db/mysql5_3308
chown mysql:mysql /opt/local/var/db/mysql5_3308
rsync -av /opt/local/var/db/mysql5/ /opt/local/var/db/mysql5_3308/

Pfoeey. Let’s see if that worked (save the below to a script file):

#!/usr/bin/env bash
# starts 3 mysql servers
sudo mysqld_safe >/opt/local/var/db/mysql5/my.out 2>&1 &
sudo mysqld_safe --defaults-file=/opt/local/etc/mysql5/my_3307.cnf \
  >/opt/local/var/db/mysql5_3307/my.out 2>&1 &
sudo mysqld_safe --defaults-file=/opt/local/etc/mysql5/my_3308.cnf \
  >/opt/local/var/db/mysql5_3308/my.out 2>&1 &

Let’s see if we got replication going:

echo "CREATE TABLE repl_test ( id int(11) PRIMARY KEY );" \
    | $my test
tail /opt/local/var/db/mysql5_3307/my.error.log
tail /opt/local/var/db/mysql5_3308/my.error.log

Setting up Memcached

Memcached needs essentially no setup. Try this (save the below to a script file):

#!/usr/bin/env bash
# starts 2 memcached servers
hostname=127.0.0.1
memperdaemon=128
daemonport1=11211
daemonport2=11212

memcached -d -l $hostname -m $memperdaemon -p $daemonport1
memcached -d -l $hostname -m $memperdaemon -p $daemonport2

Setting up Apache to use virtual hosting

We’ll have apache listen on a separate port where our app can be all by itself without getting disturbed by your other dev projects that invade its URL space.

  • mkdir ~/doodle
  • mkdir ~/doodle/htdocs
  • Edit
    /opt/local/apache2/conf/httpd.conf
    look for the line
    #Include conf/extra/httpd-vhosts.conf
    and un-comment it.
  • Then change that file:
sudo cat >/opt/local/apache2/conf/extra/httpd-vhosts.conf <<END
NameVirtualHost *:80
<VirtualHost *:80>
DocumentRoot "/opt/local/apache2/htdocs"
</VirtualHost>

Listen 8810
NameVirtualHost *:8810
<VirtualHost *:8810>
    ServerName localhost:8810
    DocumentRoot $HOME/doodle/htdocs
    <Directory "$HOME/doodle/htdocs">
      Options Indexes FollowSymLinks
      AllowOverride None
      Order allow,deny
      Allow from all
    </Directory>
</VirtualHost>
  • Test the config:
    sudo /opt/local/apache2/bin/apachectl configtest

    then start (restart if started)
    sudo /opt/local/apache2/bin/apachectl stop

    sudo /opt/local/apache2/bin/apachectl start

Make pecl/filter more pendantic

This is a pretty awkward PHP extension that implements user-submitted variable sanitization rather efficiently. Take cross-site scripting seriously, and protect against it!

  • Edit
    /opt/local/etc/php.ini
    look for the line
    [filter]

    and add
    filter.default = “special_chars”

    filter.flags = FILTER_FLAG_ENCODE_HIGH
  • Restart apache.

When are we going to do any PHP?

Let’s see if part 2 has more actual PHP in it. The above list of dependencies should give you some idea of where I’m going :-). For now try this:

cat > ~/doodle/htdocs/phpinfo.php <<END
<?php
    phpinfo();
?>
END

open http://localhost:8810/phpinfo.php

April 24, 2008

feedburner messing up my feed??

Filed under: Uncategorized — Leo Simons @ 10:18

Getting other people\'s flickr photos

I’ve tried to disable flickr consumption for now. How weird. Thanks to ColmMac for reporting :-)

April 20, 2008

Life-changing experience: Halo 3

Filed under: Uncategorized — Leo Simons @ 13:22

I’ve never really been much of a gamer, but I’m a big admirer of the underlying technology, so I bought an XBOX 360 a while ago.

I played Mass Effect, given I’m an RPG addict. Lot’s of fun, though I struggled with the controls for quite a while. I did figure it out eventually, but the final boss fight still took me about a dozen times before I succeeded.

Then I played about half of Assassin’s Creed. It looks really nice, but once I figured out the game mechanics I got bored - the game is somewhat repetitive, and the story-line is not very interesting.

So I started on Halo 3. I generally suck at these real-time shooting games, and Halo was no exception. I just couldn’t get through the third or fourth mission; I kept dying.

That is, until last weekend, when I restarted the game in easy mode. It felt a bit like admitting to myself that I’m not a Real Man, but at least I made progress. I didn’t have any time to play during the week, but I played all day (and half of the night) yesterday, and now I’ve just finished the campaign.

Wow. Truly a game of epic proportions, with a production quality easily rivalling that of a hollywood blockbuster. The experience of the final scene alone was worth the price of the console plus all the hours invested in getting there.

So, my advice to you…if you haven’t yet, go buy an xbox. They come in special Halo 3 editions.

In the meantime, I also have a question: what game could hope to impress me after Halo?

March 24, 2008

Spring+JPA == too much magic

Filed under: Uncategorized — Leo Simons @ 11:50

From The Principle of Too Much Magic (written early 2003):

Many programmers have a tendency to be Too Smart. Being Smart, they introduce various kinds of Magic into their software. This will usually lead to a software design that is more generic, dynamic and flexible than is needed, has more features than is needed, and is hence much more complex than is needed.

I learned that lesson, and I learned it really, really well. It seems that a large part of the java developer community didn’t. We were greeted with this stack trace the other day:

[2008/03/21 11:54:32.710] WARN [openjpa.Runtime]
java.lang.NullPointerException:
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:242)
at org.apache.openjpa.kernel.AbstractBrokerFactory.loadPersistentTypes(AbstractBrokerFactory.java:265)
at org.apache.openjpa.kernel.AbstractBrokerFactory.newBroker(AbstractBrokerFactory.java:197)
at org.apache.openjpa.kernel.DelegatingBrokerFactory.newBroker(DelegatingBrokerFactory.java:142)
at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:192)
at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:145)
at org.apache.openjpa.persistence.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:56)
at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean$ManagedEntityManagerFactoryInvocationHandler.invoke(AbstractEntityManagerFactoryBean.java:375)
at $Proxy11.createEntityManager(Unknown Source)
at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.createEntityManager(OpenEntityManagerInViewFilter.java:161)
at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:102)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
at org.mortbay.jetty.servlet.WebApplicationHandler$CachedChain.doFilter(WebApplicationHandler.java:821)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:471)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:568)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1530)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:636)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1482)
at org.mortbay.http.HttpServer.service(HttpServer.java:909)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:820)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:986)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:837)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:245)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
  • The servlet engine is jetty 5, the JDK is JRockit.
  • This is a spring-based web application. It uses the ContextLoaderListener to initialize spring, and an applicationContext.xml file to tell spring what beans to create.
  • Some other part of the app (not actually this servlet) uses JPA (with OpenJPA) for persistence, for which there is of course a persistence.xml.
  • Transaction management uses spring’s <tx:annotation-driven/>.

In other words, it’s pretty much how you would write a spring-and-jpa web application after reading the spring manual and the jpa manual and following their examples. Some scenario detail:

  • The stack trace happens on first servlet invocation.
  • It doesn’t always happen. It seems to happen more when the application is restarted while the cluster is under load.

The quiz question: what’s happening here?

Here’s a hint: classloaders are not normally thread-safe.

The bonus question: which one of dynamic proxies, byte code engineering, runtime aspect weaving, xml-based domain specific declarative scripting language, declarative transaction management, inversion of control, separation of concerns, reflection, classloader hacking, interceptor chaining, filter chaining, resource pooling, is not used in this scenario?

The job interview question: what would you do if you had a 9-machine cluster in production with this app on it, and then 2 of the nodes suddenly suffer from this problem on a restart?

Blog at WordPress.com.