Oracle starts at $198/year or $0.16/hour

I learned that spacewalk requires oracle the other day. To an open source weenie that sounds bad, but how much of a problem is it in practice?

Oracle XE is free, allowing up to 4GB of data, which seems like it would last you about 2 years of running spacewalk with a few thousand packages and a few hundred machines.

Oracle One costs $180 per user with a 5 user minimum, installable on one machine with two sockets (so up to 12 core for an AMD opteron). Including one year of support that adds up to about $1000.

But that’s a perpetual license. A 1-year oracle license is $36 per user, so adding in support, for one year or oracle on one machine you pay just shy of $200.

The next upgrade after oracle one is oracle standard, which costs $350 per user perpetual, or $70 per user per year.

So to get started with oracle you can use Oracle XE. When you run out of space you buy a decent database server machine and a $200 oracle license, allowing for 4 middleware nodes and one administrator. You have to pay that $200 once a year. Or you can run Oracle One in the cloud with Amazon RDS from $0.16 per hour.

When you need a resilient set up you will need oracle standard so you can cluster with Oracle RAC. You’ll need a license for each server (which can now have 4 sockets so up to 24 core), so your cost increases to 2 * 5 * 70 = $700/year.

You can also deploy oracle standard on Amazon RDS, where you’ll pay from $0.11 per hour per VM in addition to your license. Though it seems like at the moment, amazon RDS replication doesn’t work for oracle, so there’s probably no point in using that. Instead, you’ll have to switch to creating your own EBS-backed AMIs and installing oracle into them (or, perhaps, use an oracle-provided AMI).

I can find nothing that states there’s a limit to Oracle RAC scale with oracle standard. So you could potentially build big, BIG clusters this way.

$350/server/year is not exactly free of course, but this actually gets you really good clustering. That bit of capex gets you the ease of deployment and use of Oracle RAC. For some use cases, it’s probably easily the cheapest option if you take into account the opex of doing anything else. For example if Oracle Locator, the subset of Oracle Spatial that’s available in the standard edition, is good enough for you, this seems easier than figuring out your own clustered PostGIS setup. Postgres or mysql replication would’ve been just fine for spacewalk though.

Oracle Enterprise has additional features that aren’t really relevant for most scale-out (webby) stuff: using more than 4 sockets, some of the advanced administration/audit/backup tools, advanced security features, oracle spatial, compression, table partitioning, and some other bits. Enterprise still costs about a gazillion dollars.

We have to erase this idea that “oracle is ridiculously expensive” out of our brains. It just isn’t true anymore: there’s some very reasonable oracle database offerings for reasonable prices. Instead, the idea should now be “oracle enterprise is ridiculously expensive”. For things like spacewalk, where you don’t need the advanced stuff, it seems oracle is now almost always a very reasonable backend that probably can have a competitive TCO.

Creating a minimal Ubuntu Server VM

The other day I documented a kickstart process for creating base CentOS VM images. I tried but failed to get ubuntu installed using kickstart. Rather than learn the normal ubuntu process I figured I’d install manually. Beyond that, the approach is basically the same:

  • Download ubuntu server, I chose the 64-bit LTS iso (10.04).
  • create new empty VM called vanilla-ubuntu.
  • Resize the disk image to 10GB.
  • set RAM to 256MB.
  • disable just about all options including audio support, usb support, printer support, file sharing, etc.
  • set to NAT networking.
  • attach ubuntu iso to cdrom drive.
  • Accept all installer defaults, but select OpenSSH server as additional system to install
  • Log in as created user
  • run sudo passwd root to set the root password
  • sudo apt-get update && sudo apt-get -y dist-upgrade
  • to work around this bug, sudo rm /etc/motd.tail
  • sudo shutdown -h now.
  • open the settings pane, open the hard disk panel, and select “clean up disk”.
  • disconnect and remove the floppy drive. Disconnect the DVD drive.
  • (optional), open the directory containing the VM location in the terminal and mv *.vmwarevm/* . && rm -r *.vmwarevm. When you next open the VM (you will need to use File > Open), select “I moved it”.
  • Make a backup copy of the virtual machine files, this is your clean VMWare-independent base VM.
  • Start the virtual machine.
  • in the Fusion menu, select Virtual Machine > Install VMWare tools, and install the tools:
    mkdir /mnt/cdrom
    mount /dev/cdrom /mnt/cdrom
    cd /tmp
    tar zxf /mnt/cdrom/VMwareTools*.tar.gz
    ./vmware-install.pl --default
    cd
    rm -Rf /tmp/vmware-tools-distrib
    
  • Shut down the virtual machine
  • Make a backup copy of the virtual machine files, this is your clean VMWare-ready base VM.

One thing that’s easy to notice is how lightning fast ubuntu starts up. The other thing to notice is just how old CentOS / RHEL 5 really is, and IMNSHO how overdue RHEL 6 was (and CentOS 6 is):

Ubuntu 10.04 LTS CentOS 5.6
# uname -r
2.6.32-31-server
# python -V
Python 2.6.5
# perl -v
This is perl, v5.10.1 ...
# apt-cache show ruby1.9.1 | grep Version
Version: 1.9.1.378-1
# uname -r
2.6.18-238.9.1.el5
# python -V
Python 2.4.3
# perl -v
This is perl, v5.8.8 ...
# yum info ruby.x86_64 | grep Version
Version    : 1.8.5

Installing spacewalk

Spacewalk is the software behind Red Hat Network. I’ve been meaning to take a closer look for a while now. With a new release out that’s got improved cobbler integration I decided it was time to take the plunge. Having a shiny web GUI on top of cobbler seems like it could be rather neat.

Getting spacewalk installed is a bit of work since it (a) is huge, (b) needs an oracle database. But the published installation instructions worked fine for me:

Get a new Centos VM: ./make-centos32.py spacewalk. Note that Oracle express is 32 bit-only, so I decided to just do everything as a 32 bit install. Change the VM settings to use 1.5GB ram which seems like it is just about enough for development/demo purposes (specified minimum is 2GB).

Now, follow spacewalk instructions to set up Oracle Express:

$ sudo bash
# cat >>/etc/hosts <>/etc/hosts <   oracle-instantclient11.2-basic-11.2.0.2.0.i386.rpm \
>   oracle-instantclient11.2-sqlplus-11.2.0.2.0.i386.rpm

# export SWREPO=http://spacewalk.redhat.com/yum/1.4
# export EPEL=http://download.fedora.redhat.com/pub/epel
# rpm -Uvh $SWREPO/RHEL/5/i386/spacewalk-repo-1.4-1.el5.noarch.rpm
# rpm -Uvh $EPEL/5/i386/epel-release-5-4.noarch.rpm
# rpm -Uvh $SWREPO//RHEL/5/i386/spacewalk-client-repo-1.4-1.el5.noarch.rpm

# yum -y install oracle-lib-compat

# /etc/init.d/oracle-xe configure
HTTP port for Oracle Application Express: 9055
Database listener port: 1521
Password for SYS/SYSTEM: changeme12
Start at boot: y
# sqlplus 'sys/@//localhost/XE as sysdba'
SQL> create user spacewalk identified by spacewalk default tablespace users;
SQL> grant dba to spacewalk;
SQL> quit

# sqlplus spacewalk/spacewalk@//localhost/XE
SQL> alter system set processes = 400 scope=spfile; 
SQL> alter system set "_optimizer_filter_pred_pullup"=false scope=spfile; 
SQL> alter system set "_optimizer_cost_based_transformation"=off scope=spfile; 
SQL> quit  
# /sbin/service oracle-xe restart

Well that wasn’t so bad. I seem to recall installing oracle as being much harder and taking at least a day or two if you know what you’re doing. Maybe it’s just that the XE edition is so much easier, or maybe Oracle have just learned a trick or two in the 5? (wow!) years since I last touched it.

Now we’re ready to install spacewalk. The right yum repo configs were already imported, above, so we don’t need to do that again.

# yum install curl
# curl -o /tmp/gpg-key http://www.redhat.com/security/37017186.txt
# mv /tmp/gpg-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
# yum install spacewalk-oracle
...
Transaction Summary
=================================
Install     287 Package(s)
Upgrade       0 Package(s)

Total download size: 198 M
...

One of these packages results in the enabling of SELINUX. Let’s disable it again:

# echo 0 > /selinux/enforce
# vi /etc/selinux/config # change to SELINUX=disabled
# reboot

And now we can configure:

spacewalk-setup --disconnected
Database service name (SID)? XE
Username? spacewalk
Password? spacewalk
Admin Email Address? root@localhost
Should setup configure apache's default ssl server
    for you (saves original ssl.conf) [Y]? y
CA certificate password? spacewalk
Re-enter CA certificate password? spacewalk
Organization? spacewalk
Organization Unit [spacewalk.sandbox]? spacewalk
Email Address [root@localhost]? root@localhost
City? London
State? UK
Country code (Examples: "US", "JP", "IN",
    or type "?" to see a list)? GB

Now open https://spacewalk.sandbox/ in a web browser and create the admin account spacewalk/spacewalk.

Now to figure out how to make use of this beast….here’s a pretty good presentation from DevOps London about it:

Dev Ops meeting at youDevise, 3/3/2011 (Part 1 of 2) from youdevisetech on Vimeo.

Creating a minimal CentOS VM using VMWare Fusion on Mac OS X

I’ve written before about bootstrapping my development virtual infrastructure. When I did that I used an existing bare image VM I had lying around, but with CentOS 6 around the corner I wanted to document a process for creating a bare image from scratch.

First, you need CentOS 5.6 ISOs, so download CentOS if needed.

We need to create a floppy drive with a FAT or EXT2 filesystem, with a ks.cfg file on it. On the mac this seems trickier than it should be. If you have an existing linux VM you can use that. If not, make a new default VM, then

  • attach the CentOS 5.6 part1 DVD iso as the DVD drive.
  • Run installer. Settings do not matter. Disable packages to make install go quickly.
  • shut down VM.
  • echo ' ' > floppy144.flp, add floppy drive with that file.
  • start VM.

Let’s make our floppy and put a kickstart file on it:

fdformat /dev/floppy
mkfs.ext2 /dev/floppy
mount /dev/floppy /mnt/floppy

cat >/mnt/floppy/ks.cfg <<END
authconfig --enableshadow --enablemd5
bootloader --location=mbr --driveorder=sda
firewall --disable
install
cdrom
keyboard us-acentos
lang en_US.UTF-8
network --device eth0 --bootproto dhcp
rootpw changeme12
selinux --disabled
timezone --utc Europe/London
skipx
text
clearpart --all --initlabel
part /boot --fstype ext3 --size=100 --ondisk=sda
part pv.2 --size=0 --grow --ondisk=sda
volgroup VolGroup00 --pesize=32768 pv.2
logvol / --fstype ext3 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512
%packages --excludedocs --nobase
@Core
END

Note among other things the above settings disable the iptables firewall and disable selinux and use a plaintext password changeme12. You should understand what that means and be happy with the security implications before you continue.

Shut down the VM, or at least unmount the floppy. If you made that VM just to create the floppy file, you can now throw that VM away. If you’re writing your own kickstart file you may want to keep it around though in case you made a mistake, so you can go back easily and edit the file on the floppy.

Now we’re ready to start work on our image:

  • create new empty VM called vanilla-sandbox.
  • Resize the disk image to 10GB.
  • set RAM to 256MB.
  • disable just about all options including audio support, usb support, printer support, file sharing, etc.
  • set to NAT networking.
  • attach CentOS part1 DVD to cdrom drive.
  • attach floppy144.flp as a new floppy drive.
  • Start the VM.
  • At the prompt, type linux ks=floppy.
  • Wait until prompted, then press enter to reboot.
  • log in as root.
  • run yum -y update.
  • run find /var/cache -type f | xargs rm -f.
  • shutdown -h now.
  • open the settings pane, open the hard disk panel, and select “clean up disk”.
  • disconnect and remove the floppy drive. Disconnect the DVD drive.
  • (optional), open the directory containing the VM location in the terminal and mv *.vmwarevm/* . && rm -r *.vmwarevm. When you next open the VM (you will need to use File > Open), select “I moved it”.
  • Make a backup copy of the virtual machine files, this is your clean VMWare-independent base VM.
  • Start the virtual machine.
  • in the Fusion menu, select Virtual Machine > Install VMWare tools, and install the tools:
    mkdir /mnt/cdrom
    mount /dev/cdrom /mnt/cdrom
    cd /tmp
    tar zxf /mnt/cdrom/VMwareTools*.tar.gz
    yum install -y perl
    ./vmware-install.pl --default
    umount /mnt/cdrom
    cd
    rm -Rf /tmp/vmware-tools-distrib
    
  • In the Fusion menu, select Virtual Machine > CD/DVD > Disconnect CD/DVD
  • Shut down the virtual machine
  • Make a backup copy of the virtual machine files, this is your clean VMWare-ready base VM.

So now you have one squeaky clean VM image and a documented process for recreating it. Of course, most of this process isn’t particularly specific to CentOS, or VMWare, or Mac OS X. If you use different install media, these instructions work without changes for various versions of Fedora and Red Hat Enterprise Linux. According to this official ubuntu help page the same basic approach may work for recent Ubuntu releases (though replacing yum with apt-get).

I seem to remember that on linux there’s various more convenient ways to do this stuff. For example, you can install into a loopback device. But for now, this will do.

Using libvirt with VMWare Fusion on Mac OS X

Even though its website doesn’t say so, recent versions of libvirt should work reasonably well on Mac OS X now. Let’s give it a shot at trying to get it to work with VMWare Fusion.

Now the fun starts:

$ virsh -c vmwarews:///session list
error: internal error vmrun utility is missing
error: failed to connect to the hypervisor
$ export PATH=/Library/Application\ Support/VMware\ Fusion/:$PATH
$ virsh -c vmwarews:///session list
error: Cannot find 'vmware' in path: No such file or directory
error: failed to connect to the hypervisor
$ export PATH=/Applications/VMware\ Fusion.app/Contents/MacOS:$PATH
$ virsh -c vmwarews:///session list

This starts the VMWare Fusion graphical interface. Not what we want. libvirt expects to run vmware -v to get the version, but that doesn’t work on the mac. Let’s hack the code:

int
vmwareExtractVersion(struct vmware_driver *driver)
{
    unsigned long version = 0;
    // hardcode version to 7.1 to try to trick libvirt into thinking we have workstation 7.1
    // char *tmp;
    const char *tmp = "7.1"; 
    int ret = -1;
    // virCommandPtr cmd;
    // char * outbuf = NULL;
    // const char * bin = (driver->type == TYPE_PLAYER) ? "vmplayer" : "vmware";
    // const char * pattern = (driver->type == TYPE_PLAYER) ?
    //             "VMware Player " : "VMware Workstation ";

    // cmd = virCommandNewArgList(bin, "-v", NULL);
    // virCommandSetOutputBuffer(cmd, &outbuf);
    // 
    // if (virCommandRun(cmd, NULL) < 0)
    //     goto cleanup;
    // 
    // if ((tmp = STRSKIP(outbuf, pattern)) == NULL) {
    //     vmwareError(VIR_ERR_INTERNAL_ERROR,
    //                 _("failed to parse %s version"), bin);
    //     goto cleanup;
    // }
    // 
    if (virParseVersionString(tmp, &version) version = version;
    ret = 0;

cleanup:
    // virCommandFree(cmd);
    // VIR_FREE(outbuf);
    return ret;
}

Recompile, reinstall, then

$ virsh -c vmwarews:///session list
error: internal error No yet handled value 'nat' for VMX entry 'ethernet0.connectionType'
error: failed to connect to the hyperviso

Looking at the code:

if (connectionType == NULL || STRCASEEQ(connectionType, "bridged")) {
    // set up ...
} else if (STRCASEEQ(connectionType, "hostonly")) {
    // FIXME
    VMX_ERROR(...); ...
} else if (STRCASEEQ(connectionType, "nat")) {
    // FIXME
    VMX_ERROR(...); ...
} else if (STRCASEEQ(connectionType, "custom")) {
   // set up ... (treat like bridged)
} else {
   VMX_ERROR(...); ...
}

It’s obvious a bridged network is what is supported. So let’s change all the VMs to use bridged networking….

$ virsh -c vmwarews:///session list
error: internal error Missing essential config entry 'ethernet0.networkName'
error: failed to connect to the hypervisor

cat >> VMXFILE <<END
ethernet0.networkName = ""
ethernet1.networkName = ""
END
$ virsh -c vmwarews:///session list
 Id Name                 State
----------------------------------
48627 VMNAME             running

$ virsh -c vmwarews:///session dominfo 48627
Id:             48627
Name:           VMNAME
UUID:           564d2021-a5e3-a0b2-8f55-0258afabbbec
OS Type:        hvm
State:          running
CPU(s):         1


Max memory:     1048576 kB
Used memory:    1048576 kB
Persistent:     yes

So it seems like with the above hacks, you can use libvirt with VMWare Fusion on Mac OS X.

On the other hand, I cannot get VirtualBox (4.0.8) working:

$ virsh -d 5 -c vbox:///session list
commands: "list"
(hangs at 100% CPU forever)
^C