Posts Tagged ‘ec2’

Amazon EC2: Watch your Disk Space!

Wednesday, February 24th, 2010

Today at imVOX we suffered a short outage and downtime of our main service and to our embedded widget due to our database going down. We had everything back up within 15 minutes.

The cause of the downtime was that we ran out of disk space on our database server. We found our when our always vigilant Pro user Rev Lazaro (@revlazaro on Twitter) let us know that his group was unable to log into imVOX via Twitter.

We run our Oracle database on Amazon’s EC2 service, using their Elastic Block Storage (EBS) for storage on the database. We use four 100GB EBS units in a RAID configuration, keeping one for parity which gives us 300GB of usable space for our database. This can easily scale up size-wise and is perfect for our database size. They are also monitored by Amazon’s Cloudwatch service, letting us know instantly if something goes wrong or we run out of space.

So how did we run out of space? Log files.

The EC2 instance only had 10GB of storage which is normally enough. Cloudwatch isn’t watching the EC2 instance for space and the log files from the database transactions got rather large and filled the disk. This instantly locked up the database from executing additional transactions- halting the imVOX service completely.

Thankfully we had a user notice quickly and we were able to move the log files and restart the database without issue.

Lessons Learned:

  1. Database logs can get big
  2. Just because your database has space, doesn’t mean your db server does
  3. Cloudwatch might not catch something like your EC2 instance running out of space
  • Share/Bookmark

Ask imVOX: Where should I host my guild?

Friday, January 8th, 2010
We use Amazon AWS services (http://aws.amazon.com/) for hosting with their EC2 platform as our primary web and voice servers (http://aws.amazon.com/ec2/). However it is quite pricy at approximately $100/month to get started and probably way overkill for most guilds except super large ones like Fires of Heaven or similar. With my knowledge about your size- I wouldn’t use it for what you’ve got yet.
There’s a few other really great market options. I’ve had a great deal of experience with Slicehost (http://www.slicehost.com/) in hosting my own personal blog and other websites. It gives you full access to your own virtual private server (VPS) which is almost never effected by other people using it. The 256 or 512 slice is sufficient for most people’s needs and I run 3 blogs on a 256 slice. It is more expensive than your current service slightly but is one of the best purchases I’ve ever made. The 256 slice is $20/month and the 512 slice is $38/month.
A cheaper option that gives you more, but is a similar service is called Linode (http://www.linode.com/). They too offer VPS hosting and by some claims are faster than Slicehost and cheaper overall. You geta  360MB slice for $20/month which is a nice deal. I’ve had serveral friends use it and speak well of it even for fairly large dynamic sites like InstantWatcher (http://instantwatcher.com) which I believe is a Rails app.
Shared hosting offerings are always a tricky proposition. Companies like Dreamhost, Bluehost and A Small Orange offer it for very low prices. I’ve personally found the stability of these to be low and the customer support to be substandard (at least for Dreamhost which I used for a year, I can’t speak to the others). You couldn’t pay me to go back to them.
Another thing hosting-wise I might consider is buying your own hardware server and finding somewhere to put it. Hardware these days is dirt cheap and there’s always some good deal on eBay or Craigslist to be had. You don’t need much for most websites as long as they are well written. Finding somewhere to put it is the hard part. One place I might start your search is at a local university. They generally have a ton of extra rackspace, plenty of extra bandwidth and no one notices an extra server. At some places like MIT every computer actually gets a ‘real IP’ on the internet and there are literally dozens of servers for every floor of many dorms. If you’ve got a friend there, I’m almost certain you could throw your server in there and no one would notice or mess with it too much. Of course play nice on their network and maybe make a donation to the university in lieu of hosting payments. I’ve got 6 servers racked at a Boston-area university like this and it works great. If you go the commercial route for hosting your rack server you’ll pay dearly for it, starting around $200/month per server from what I’ve seen.
My best recommendation if you can’t find somewhere cheap to install a server, is to use a VPS host like Slicehost. Backup your database frequently (using a cron job), and check everything into a private Github (http://github.com) repo so if you need to revert your website or move to a new server then the migration will take only an hour or so. Download your database backups and burn them to CDs or keep them on a flash drive.

An imVOX member recently asked me to recommend a good host for their guild site. There’s a lot of bad hosting out there and they had recently had a poor experience with one. They were curious of which service imVOX uses for hosting as well. Here’s my best advice for hosting based on my personal experiences.

We use Amazon AWS for hosting with their EC2 platform as our web and voice servers. However it is quite pricy at approximately $100/month to get started and probably way overkill for most guilds except super large ones like Fires of Heaven or Elitist Jerks (neither of which actually use EC2. To the best of my knowledge they are hosted in a Cambridge-area facility on ‘real’ hardware). With my knowledge about most guild’s size- I wouldn’t use AWS for what you’ve got yet.

There’s a few other really great market options. I’ve had a great deal of experience with Slicehost in hosting my own personal blog and other websites. It gives you full access to your own virtual private server (VPS) which is almost never effected by other people using the service. The 256 or 512 slice is sufficient for most people’s needs and I run 3 blogs on a 256 slice. It is more expensive than your current service slightly but is one of the best purchases I’ve ever made. The 256 slice is $20/month and the 512 slice is $38/month.

Another VPS that gives you more for your money is called Linode. By some claims are faster than Slicehost and cheaper when counted by memory. You geta  360MB slice for $20/month which is a nice deal. I’ve had serveral friends use it and speak well of it even for fairly large dynamic sites like InstantWatcher which I believe is a Rails app.

Shared hosting offerings are always a tricky proposition. Companies like Dreamhost, Bluehost and A Small Orange offer it for very low prices. I’ve personally found the stability of these to be low and the customer support to be substandard (at least for Dreamhost which I used for a year, I can’t speak to the others). You couldn’t pay me to go back to them.

Another thing hosting-wise I might consider is buying your own hardware server and finding somewhere to put it. Hardware these days is dirt cheap and there’s always some good deal on eBay or Craigslist to be had. You don’t need much for most websites as long as they are well written. Finding somewhere to put it is the hard part. One place I might start your search is at a local university. They generally have a ton of extra rackspace, plenty of extra bandwidth and no one notices an extra server. At some places like MIT every computer actually gets a ‘real IP’ on the internet and there are literally dozens of servers for every floor of many dorms. If you’ve got a friend there, I’m almost certain you could throw your server in there and no one would notice or mess with it too much. Of course play nice on their network and maybe make a donation to the university in lieu of hosting payments. I’ve got 6 servers racked at a local university with the consent of the administration and it works out great. If you go the commercial route for hosting your rack server you’ll pay dearly for it, starting around $200/month per server from what I’ve seen.

My best recommendation if you can’t find somewhere cheap to install a server, is to use a VPS host like Slicehost. Backup your database frequently (using a cron job), and check everything into a private Github repo so if you need to revert your website or move to a new server then the migration will take only an hour or so. Download your database backups and burn them to CDs or keep them on a flash drive, or better yet- backup to other servers or a service like Mozy.

Disclaimer: imVOX has no ownership, management or stake in any of the mentioned companies here. We haven’t been paid to blog about them and aside from standard customer support emails we haven’t been in touch with them either. These are just our (David’s) opinions on service based on personal experience.

  • Share/Bookmark