Oracle starts at $198/year or $0.16/hour

I learned that spacewalk requires oracle the other day. To an open source weenie that sounds bad, but how much of a problem is it in practice?

Oracle XE is free, allowing up to 4GB of data, which seems like it would last you about 2 years of running spacewalk with a few thousand packages and a few hundred machines.

Oracle One costs $180 per user with a 5 user minimum, installable on one machine with two sockets (so up to 12 core for an AMD opteron). Including one year of support that adds up to about $1000.

But that’s a perpetual license. A 1-year oracle license is $36 per user, so adding in support, for one year or oracle on one machine you pay just shy of $200.

The next upgrade after oracle one is oracle standard, which costs $350 per user perpetual, or $70 per user per year.

So to get started with oracle you can use Oracle XE. When you run out of space you buy a decent database server machine and a $200 oracle license, allowing for 4 middleware nodes and one administrator. You have to pay that $200 once a year. Or you can run Oracle One in the cloud with Amazon RDS from $0.16 per hour.

When you need a resilient set up you will need oracle standard so you can cluster with Oracle RAC. You’ll need a license for each server (which can now have 4 sockets so up to 24 core), so your cost increases to 2 * 5 * 70 = $700/year.

You can also deploy oracle standard on Amazon RDS, where you’ll pay from $0.11 per hour per VM in addition to your license. Though it seems like at the moment, amazon RDS replication doesn’t work for oracle, so there’s probably no point in using that. Instead, you’ll have to switch to creating your own EBS-backed AMIs and installing oracle into them (or, perhaps, use an oracle-provided AMI).

I can find nothing that states there’s a limit to Oracle RAC scale with oracle standard. So you could potentially build big, BIG clusters this way.

$350/server/year is not exactly free of course, but this actually gets you really good clustering. That bit of capex gets you the ease of deployment and use of Oracle RAC. For some use cases, it’s probably easily the cheapest option if you take into account the opex of doing anything else. For example if Oracle Locator, the subset of Oracle Spatial that’s available in the standard edition, is good enough for you, this seems easier than figuring out your own clustered PostGIS setup. Postgres or mysql replication would’ve been just fine for spacewalk though.

Oracle Enterprise has additional features that aren’t really relevant for most scale-out (webby) stuff: using more than 4 sockets, some of the advanced administration/audit/backup tools, advanced security features, oracle spatial, compression, table partitioning, and some other bits. Enterprise still costs about a gazillion dollars.

We have to erase this idea that “oracle is ridiculously expensive” out of our brains. It just isn’t true anymore: there’s some very reasonable oracle database offerings for reasonable prices. Instead, the idea should now be “oracle enterprise is ridiculously expensive”. For things like spacewalk, where you don’t need the advanced stuff, it seems oracle is now almost always a very reasonable backend that probably can have a competitive TCO.

Creating a minimal Ubuntu Server VM

The other day I documented a kickstart process for creating base CentOS VM images. I tried but failed to get ubuntu installed using kickstart. Rather than learn the normal ubuntu process I figured I’d install manually. Beyond that, the approach is basically the same:

  • Download ubuntu server, I chose the 64-bit LTS iso (10.04).
  • create new empty VM called vanilla-ubuntu.
  • Resize the disk image to 10GB.
  • set RAM to 256MB.
  • disable just about all options including audio support, usb support, printer support, file sharing, etc.
  • set to NAT networking.
  • attach ubuntu iso to cdrom drive.
  • Accept all installer defaults, but select OpenSSH server as additional system to install
  • Log in as created user
  • run sudo passwd root to set the root password
  • sudo apt-get update && sudo apt-get -y dist-upgrade
  • to work around this bug, sudo rm /etc/motd.tail
  • sudo shutdown -h now.
  • open the settings pane, open the hard disk panel, and select “clean up disk”.
  • disconnect and remove the floppy drive. Disconnect the DVD drive.
  • (optional), open the directory containing the VM location in the terminal and mv *.vmwarevm/* . && rm -r *.vmwarevm. When you next open the VM (you will need to use File > Open), select “I moved it”.
  • Make a backup copy of the virtual machine files, this is your clean VMWare-independent base VM.
  • Start the virtual machine.
  • in the Fusion menu, select Virtual Machine > Install VMWare tools, and install the tools:
    mkdir /mnt/cdrom
    mount /dev/cdrom /mnt/cdrom
    cd /tmp
    tar zxf /mnt/cdrom/VMwareTools*.tar.gz
    ./vmware-install.pl --default
    cd
    rm -Rf /tmp/vmware-tools-distrib
    
  • Shut down the virtual machine
  • Make a backup copy of the virtual machine files, this is your clean VMWare-ready base VM.

One thing that’s easy to notice is how lightning fast ubuntu starts up. The other thing to notice is just how old CentOS / RHEL 5 really is, and IMNSHO how overdue RHEL 6 was (and CentOS 6 is):

Ubuntu 10.04 LTS CentOS 5.6
# uname -r
2.6.32-31-server
# python -V
Python 2.6.5
# perl -v
This is perl, v5.10.1 ...
# apt-cache show ruby1.9.1 | grep Version
Version: 1.9.1.378-1
# uname -r
2.6.18-238.9.1.el5
# python -V
Python 2.4.3
# perl -v
This is perl, v5.8.8 ...
# yum info ruby.x86_64 | grep Version
Version    : 1.8.5

Installing spacewalk

Spacewalk is the software behind Red Hat Network. I’ve been meaning to take a closer look for a while now. With a new release out that’s got improved cobbler integration I decided it was time to take the plunge. Having a shiny web GUI on top of cobbler seems like it could be rather neat.

Getting spacewalk installed is a bit of work since it (a) is huge, (b) needs an oracle database. But the published installation instructions worked fine for me:

Get a new Centos VM: ./make-centos32.py spacewalk. Note that Oracle express is 32 bit-only, so I decided to just do everything as a 32 bit install. Change the VM settings to use 1.5GB ram which seems like it is just about enough for development/demo purposes (specified minimum is 2GB).

Now, follow spacewalk instructions to set up Oracle Express:

$ sudo bash
# cat >>/etc/hosts <>/etc/hosts <   oracle-instantclient11.2-basic-11.2.0.2.0.i386.rpm \
>   oracle-instantclient11.2-sqlplus-11.2.0.2.0.i386.rpm

# export SWREPO=http://spacewalk.redhat.com/yum/1.4
# export EPEL=http://download.fedora.redhat.com/pub/epel
# rpm -Uvh $SWREPO/RHEL/5/i386/spacewalk-repo-1.4-1.el5.noarch.rpm
# rpm -Uvh $EPEL/5/i386/epel-release-5-4.noarch.rpm
# rpm -Uvh $SWREPO//RHEL/5/i386/spacewalk-client-repo-1.4-1.el5.noarch.rpm

# yum -y install oracle-lib-compat

# /etc/init.d/oracle-xe configure
HTTP port for Oracle Application Express: 9055
Database listener port: 1521
Password for SYS/SYSTEM: changeme12
Start at boot: y
# sqlplus 'sys/@//localhost/XE as sysdba'
SQL> create user spacewalk identified by spacewalk default tablespace users;
SQL> grant dba to spacewalk;
SQL> quit

# sqlplus spacewalk/spacewalk@//localhost/XE
SQL> alter system set processes = 400 scope=spfile; 
SQL> alter system set "_optimizer_filter_pred_pullup"=false scope=spfile; 
SQL> alter system set "_optimizer_cost_based_transformation"=off scope=spfile; 
SQL> quit  
# /sbin/service oracle-xe restart

Well that wasn’t so bad. I seem to recall installing oracle as being much harder and taking at least a day or two if you know what you’re doing. Maybe it’s just that the XE edition is so much easier, or maybe Oracle have just learned a trick or two in the 5? (wow!) years since I last touched it.

Now we’re ready to install spacewalk. The right yum repo configs were already imported, above, so we don’t need to do that again.

# yum install curl
# curl -o /tmp/gpg-key http://www.redhat.com/security/37017186.txt
# mv /tmp/gpg-key /etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
# yum install spacewalk-oracle
...
Transaction Summary
=================================
Install     287 Package(s)
Upgrade       0 Package(s)

Total download size: 198 M
...

One of these packages results in the enabling of SELINUX. Let’s disable it again:

# echo 0 > /selinux/enforce
# vi /etc/selinux/config # change to SELINUX=disabled
# reboot

And now we can configure:

spacewalk-setup --disconnected
Database service name (SID)? XE
Username? spacewalk
Password? spacewalk
Admin Email Address? root@localhost
Should setup configure apache's default ssl server
    for you (saves original ssl.conf) [Y]? y
CA certificate password? spacewalk
Re-enter CA certificate password? spacewalk
Organization? spacewalk
Organization Unit [spacewalk.sandbox]? spacewalk
Email Address [root@localhost]? root@localhost
City? London
State? UK
Country code (Examples: "US", "JP", "IN",
    or type "?" to see a list)? GB

Now open https://spacewalk.sandbox/ in a web browser and create the admin account spacewalk/spacewalk.

Now to figure out how to make use of this beast….here’s a pretty good presentation from DevOps London about it:

Dev Ops meeting at youDevise, 3/3/2011 (Part 1 of 2) from youdevisetech on Vimeo.

Creating a minimal CentOS VM using VMWare Fusion on Mac OS X

I’ve written before about bootstrapping my development virtual infrastructure. When I did that I used an existing bare image VM I had lying around, but with CentOS 6 around the corner I wanted to document a process for creating a bare image from scratch.

First, you need CentOS 5.6 ISOs, so download CentOS if needed.

We need to create a floppy drive with a FAT or EXT2 filesystem, with a ks.cfg file on it. On the mac this seems trickier than it should be. If you have an existing linux VM you can use that. If not, make a new default VM, then

  • attach the CentOS 5.6 part1 DVD iso as the DVD drive.
  • Run installer. Settings do not matter. Disable packages to make install go quickly.
  • shut down VM.
  • echo ' ' > floppy144.flp, add floppy drive with that file.
  • start VM.

Let’s make our floppy and put a kickstart file on it:

fdformat /dev/floppy
mkfs.ext2 /dev/floppy
mount /dev/floppy /mnt/floppy

cat >/mnt/floppy/ks.cfg <<END
authconfig --enableshadow --enablemd5
bootloader --location=mbr --driveorder=sda
firewall --disable
install
cdrom
keyboard us-acentos
lang en_US.UTF-8
network --device eth0 --bootproto dhcp
rootpw changeme12
selinux --disabled
timezone --utc Europe/London
skipx
text
clearpart --all --initlabel
part /boot --fstype ext3 --size=100 --ondisk=sda
part pv.2 --size=0 --grow --ondisk=sda
volgroup VolGroup00 --pesize=32768 pv.2
logvol / --fstype ext3 --name=LogVol00 --vgname=VolGroup00 --size=1024 --grow
logvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512
%packages --excludedocs --nobase
@Core
END

Note among other things the above settings disable the iptables firewall and disable selinux and use a plaintext password changeme12. You should understand what that means and be happy with the security implications before you continue.

Shut down the VM, or at least unmount the floppy. If you made that VM just to create the floppy file, you can now throw that VM away. If you’re writing your own kickstart file you may want to keep it around though in case you made a mistake, so you can go back easily and edit the file on the floppy.

Now we’re ready to start work on our image:

  • create new empty VM called vanilla-sandbox.
  • Resize the disk image to 10GB.
  • set RAM to 256MB.
  • disable just about all options including audio support, usb support, printer support, file sharing, etc.
  • set to NAT networking.
  • attach CentOS part1 DVD to cdrom drive.
  • attach floppy144.flp as a new floppy drive.
  • Start the VM.
  • At the prompt, type linux ks=floppy.
  • Wait until prompted, then press enter to reboot.
  • log in as root.
  • run yum -y update.
  • run find /var/cache -type f | xargs rm -f.
  • shutdown -h now.
  • open the settings pane, open the hard disk panel, and select “clean up disk”.
  • disconnect and remove the floppy drive. Disconnect the DVD drive.
  • (optional), open the directory containing the VM location in the terminal and mv *.vmwarevm/* . && rm -r *.vmwarevm. When you next open the VM (you will need to use File > Open), select “I moved it”.
  • Make a backup copy of the virtual machine files, this is your clean VMWare-independent base VM.
  • Start the virtual machine.
  • in the Fusion menu, select Virtual Machine > Install VMWare tools, and install the tools:
    mkdir /mnt/cdrom
    mount /dev/cdrom /mnt/cdrom
    cd /tmp
    tar zxf /mnt/cdrom/VMwareTools*.tar.gz
    yum install -y perl
    ./vmware-install.pl --default
    umount /mnt/cdrom
    cd
    rm -Rf /tmp/vmware-tools-distrib
    
  • In the Fusion menu, select Virtual Machine > CD/DVD > Disconnect CD/DVD
  • Shut down the virtual machine
  • Make a backup copy of the virtual machine files, this is your clean VMWare-ready base VM.

So now you have one squeaky clean VM image and a documented process for recreating it. Of course, most of this process isn’t particularly specific to CentOS, or VMWare, or Mac OS X. If you use different install media, these instructions work without changes for various versions of Fedora and Red Hat Enterprise Linux. According to this official ubuntu help page the same basic approach may work for recent Ubuntu releases (though replacing yum with apt-get).

I seem to remember that on linux there’s various more convenient ways to do this stuff. For example, you can install into a loopback device. But for now, this will do.

Using libvirt with VMWare Fusion on Mac OS X

Even though its website doesn’t say so, recent versions of libvirt should work reasonably well on Mac OS X now. Let’s give it a shot at trying to get it to work with VMWare Fusion.

Now the fun starts:

$ virsh -c vmwarews:///session list
error: internal error vmrun utility is missing
error: failed to connect to the hypervisor
$ export PATH=/Library/Application\ Support/VMware\ Fusion/:$PATH
$ virsh -c vmwarews:///session list
error: Cannot find 'vmware' in path: No such file or directory
error: failed to connect to the hypervisor
$ export PATH=/Applications/VMware\ Fusion.app/Contents/MacOS:$PATH
$ virsh -c vmwarews:///session list

This starts the VMWare Fusion graphical interface. Not what we want. libvirt expects to run vmware -v to get the version, but that doesn’t work on the mac. Let’s hack the code:

int
vmwareExtractVersion(struct vmware_driver *driver)
{
    unsigned long version = 0;
    // hardcode version to 7.1 to try to trick libvirt into thinking we have workstation 7.1
    // char *tmp;
    const char *tmp = "7.1"; 
    int ret = -1;
    // virCommandPtr cmd;
    // char * outbuf = NULL;
    // const char * bin = (driver->type == TYPE_PLAYER) ? "vmplayer" : "vmware";
    // const char * pattern = (driver->type == TYPE_PLAYER) ?
    //             "VMware Player " : "VMware Workstation ";

    // cmd = virCommandNewArgList(bin, "-v", NULL);
    // virCommandSetOutputBuffer(cmd, &outbuf);
    // 
    // if (virCommandRun(cmd, NULL) < 0)
    //     goto cleanup;
    // 
    // if ((tmp = STRSKIP(outbuf, pattern)) == NULL) {
    //     vmwareError(VIR_ERR_INTERNAL_ERROR,
    //                 _("failed to parse %s version"), bin);
    //     goto cleanup;
    // }
    // 
    if (virParseVersionString(tmp, &version) version = version;
    ret = 0;

cleanup:
    // virCommandFree(cmd);
    // VIR_FREE(outbuf);
    return ret;
}

Recompile, reinstall, then

$ virsh -c vmwarews:///session list
error: internal error No yet handled value 'nat' for VMX entry 'ethernet0.connectionType'
error: failed to connect to the hyperviso

Looking at the code:

if (connectionType == NULL || STRCASEEQ(connectionType, "bridged")) {
    // set up ...
} else if (STRCASEEQ(connectionType, "hostonly")) {
    // FIXME
    VMX_ERROR(...); ...
} else if (STRCASEEQ(connectionType, "nat")) {
    // FIXME
    VMX_ERROR(...); ...
} else if (STRCASEEQ(connectionType, "custom")) {
   // set up ... (treat like bridged)
} else {
   VMX_ERROR(...); ...
}

It’s obvious a bridged network is what is supported. So let’s change all the VMs to use bridged networking….

$ virsh -c vmwarews:///session list
error: internal error Missing essential config entry 'ethernet0.networkName'
error: failed to connect to the hypervisor

cat >> VMXFILE <<END
ethernet0.networkName = ""
ethernet1.networkName = ""
END
$ virsh -c vmwarews:///session list
 Id Name                 State
----------------------------------
48627 VMNAME             running

$ virsh -c vmwarews:///session dominfo 48627
Id:             48627
Name:           VMNAME
UUID:           564d2021-a5e3-a0b2-8f55-0258afabbbec
OS Type:        hvm
State:          running
CPU(s):         1


Max memory:     1048576 kB
Used memory:    1048576 kB
Persistent:     yes

So it seems like with the above hacks, you can use libvirt with VMWare Fusion on Mac OS X.

On the other hand, I cannot get VirtualBox (4.0.8) working:

$ virsh -d 5 -c vbox:///session list
commands: "list"
(hangs at 100% CPU forever)
^C

Installing VMWare Studio on VMware Fusion on Mac OS X

VMWare Studio is a free virtual appliance from VMware that allows you to “author, configure, deploy and customize virtual machines”. There’s no instructions how to get it running on the mac, but it works just fine with VMWare Fusion:

  • Register with VMWare
  • (Buy), download and install VMWare Fusion
  • Download and install VMWare OVF tool
  • Download VMWare Studio VMDK disk, unzip
  • Download VMWare Studio OVF 1.0 spec file
  • Convert VMware Studio OVF to VMX: /opt/vmware/ovftool/ovftool -tt=VMX VMware_Studio-2.5.0.0-387333_OVF10.ovf VMware_Studio-2.5.0.0-387333_VMX.vmx
  • Import VMWare Studio VMX into VMWare Fusion

Start the VMware Studio VM. Read and agree to the license agreement. Set a root password. Open the provided URL in your web browser.

$ ssh 192.168.1.75
root@localhost:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 8.04.4 LTS
Release:        8.04
Codename:       hardy

VMWare Studio looks like it consists of lighttpd serving up a GWT frontend that talks to a sfcb CIM backend, where CIM is a standard that’s part of the bigger Web-based Enterprise Management standard. Presumably they’re using the same CIM setup as, say, vSphere. Pretty cool setup.

I’ve built a few VMs and it all seems to work ok. The one gotcha is setting the build settings when defining VM profiles. Pick VMWare Workstation, and point it at /Library/Application Support/VMware Fusion to use the vmrun in VMWare Fusion.

I’ve also tried importing the OVF into VirtualBox which also works. So if you have a VMware install somewhere else (I think even VMWare Player would work since it provides a vmrun command) you could avoid buying Fusion.

I don’t think I would actually want to use VMWare Studio for building linux virtual machines; the scripted approach of cobbler seems preferable. OTOH, VMWare Studio provides good windows support, and can be used to create windows VMs from scratch. I can imagine:

  • define a windows build in VMWare Studio that outputs OVF
  • write a script that interacts with VMWare Studio through CIM to create new windows virtual machines
  • load those windows virtual machines into your hypervisor of choice if it supports OVF
  • if it does not support OVF but it supports VMDK/VMX, use VWWare OVFtool to convert to VMDK/VMX, then load into your hypervisor of choice (for example use qemu-img convert or import into amazon)

That seems a considerably lighter weight approach than going with Microsoft System Center Configuration Manager 2007 Operating System Deployment (Microsoft’s name for its cobbler). Interesting. Then again, if you’re managing a lot of windows host you may be invested in SCCM already?

Getting started with vagrant on Mac OS X

Vagrant wants an up-to-date RubyGems. From experience changing the ruby install bundled with Mac OS X can be a bad idea. I considered using rvm but I don’t think I need all that power here. So I’ll be my usual old-fashioned self, and install my own ruby in /usr/local.

  • Download latest ruby source code (1.9.2-p180 at time of writing), ./configure && make && make install (this results in a 64-bit only build)
  • create a new terminal, check gem environment gives the new ruby, run gem update --system && gem update
  • if you now get a lot of deprecation warnings, follow this advice to get rid of them: gem pristine --all --no-extensions (there are no native extension installed yet so no need to worry about those)
  • if you get any instructions about running rdoc-data, ignore the instructions, according to this bug report it’s not needed for ruby 1.9.2.

So now we have up-to-date ruby and up-to-date RubyGems. Good. Let’s get virtualbox and vagrant:

Get started:

vagrant box add lucid32 http://files.vagrantup.com/lucid32.box
mkdir test1
cd test1
vagrant init
vi Vagrantfile
# change config.vm.box to "lucid32"
# set config.ssh.max_tries to something high, say 50000
# set config.ssh.timeout to something high, say 3000000
vagrant up
vagrant ssh
echo test
exit

Works for me.

My preferred hypervisor on the mac is VMWare Fusion though. I already have a bunch of custom scripts set up to create and manage VMs and puppet them up. I could migrate those over to VirtualBox I guess and then change all my scripts to use vagrant, but I don’t really see that it offers me any benefit right now. Definitely a useful tool though and it’ll be interesting to follow its development.