A week after going live with Oracle Mix, we’ve gotten a considerable amount of buzz and interest from every aspect of Mix. Most everyone we’ve talked with are really happy to see Oracle becoming more transparent. Mix is a huge part of that. There are a lot of tool heads out there who are really interested in how we made it happen. So, here it is… the story of how Mix was conceived and built.
Those who have kept up with this blog, know about the internal social network we created inside the walls of Oracle; we call that Connect. For those who don’t know anything about Connect, it’s basically a skunk works project that became really popular — you can read more about it here. Connect’s success has opened the minds of many people within Oracle. Many people know the potential of the community and network effects that we’re seeing with the large social networks, but it really hasn’t been applied to enterprise communities yet. Several people at Oracle recognized this and so about 6 weeks leading up to Open World, the Mix project was approved by a few senior Oracle execs and hence was born. Paul, kicked it off and told us we had 6 weeks to have a working application by Open World. I almost had a conniption at that point. Connect took about 6 weeks for me to build (by myself), but to build a web application that had more features, be publicly facing, and had to run on a tech stack (our own) that we had no idea would work well, was just crazy. We didn’t have hardware, data center space, and had a ton of Oracle security, legal and other hurdles to work through — not including development resources. I knew at that point that we needed help to pull it together. Luckily, I knew about people like Ola Bini and ThoughtWorks. I knew that they were doing lots of work around JRuby, and I knew that they practice preach agile development practices. So, we met immediately. It was a perfect match.
The team was comprised of four developers from TW (Ola Bini, Alexey Verkhovsky, Toby Tripp, and Matt Wastrodowski) and me from the Oracle side. We also had a project manager (Mike Royle) who whipped us into shape at the end of every iteration. We started coding about 3.5 to 4 weeks before Open World. At that point, we still didn’t have hardware and there were a ton of unknowns surrounding JRuby running on the Oracle AppServer (and if it did, what performance would be like). We bit the bullet and went for it. Rails was an essential ingredient to this project. I don’t know of other frameworks that would have enabled us to build this particular application so rapidly. The beauty of Rails is the plugin system. We leveraged a whole slew of plugins that made tagging, comments, ratings, paginations, search, authentication, and other common features a simple feature to implement and manage.
We built most of Mix using the standard Ruby interpreter (aka, MRI — Matz’ Ruby Interpreter) on Oracle DB XE using Ruby-OCI. Developing Rails apps is still much easier using MRI because JRuby takes a while to startup. While developing on the Oracle DB, we encountered several issues with the Ruby-OCI adapter (the Ruby-OCI adapter is in need of some official Oracle support… more on that later). While most of the team was using MRI/OCI, I was working on getting hardware and configuring what we had. Once we finally got hardware (4 x 2 cpu-single-core, 12 gb mem, small HDs — yeah… crappy hardware that was on its way to being decommissioned), I spent several days working with the sys admins to get one box set up with Oracle DB 10g (I would have gone with 11g, but the DBAs who were going to support us weren’t up to speed on the configuration of 11g yet) and the other 3 boxes with Oracle Application Server 10g. The 3 app servers were configured to be load balanced by a BigIP layer 7 based load balancer. I set up jRuby and the necessary pieces to allow us to deploy easily on Oracle App Server… namely installing admin_client.jar, which is a CLI for managing the Oracle App Server.
One critical part of the project was to support Oracle’s SSO policy. Meaning every web app that we deploy has to support SSO using the Oracle SSO service. It took me a few days to figure out how all this worked but once we did, it wasn’t difficult to plug it into the acts_as_authenticated Rails plugin that we were using. If anyone is looking to do something like this, leave a comment and I’ll send you the code — maybe I’ll release it as an acts_as_authenticated_through_oracle_sso plugin if there are enough people doing this sort of thing.
After I had the boxes primed, we started doing early deploys on JRuby. The performance was terrible. About 20-40 reqs/sec on a single app server. Turns out that I didn’t have some of the production settings configured properly (i.e., not caching ruby classes, etc.) Once I modified a few environment settings (standard rails prod settings), I got about 80 reqs/sec on a single app server… better but not enough. At the same time, Ola and the JRuby team found some interesting bottlenecks in the JRuby code. Within in a day or two, Ola and team had them patched up and we were beginning to see around 150-200 reqs/sec. After the app server warmed up, things got real interesting… the numbers went way up (400-600 reqs/sec). *Disclaimer: I’m not a performance guru. I did most of my tests using Apache Bench running against the logged in home page of Mix, which btw, doesn’t use any fragment caching at all (actually, Mix is not making use of any caching right now — we’ll get to it eventually, but it doesn’t seem like we need it yet).
The 3.5 to 4 weeks leading up to Open World was nuts. I spent a ton of time taking care of deployment/hardware issues rather than coding. The rest of the team worked remotely (Toby and Matt in Chicago, Alexey in Calgary, Ola in London) which is sort of against the agile mantra, but it was done to cut cost. We communicated through Skype’s bookmarked chat feature which is an excellent tool if you need to work as an agile team remotely. The week before Open World, Toby, Matt and I rendezvoused in San Francisco and knocked out a ton of functionality. Even though we had a productive week leading up to Open World, we knew that the app would still need a lot of polish past go-live, but it’s amazing how much a small team working mostly remotely could accomplish in that short period of time.
This was an amazing project to be a part of. And one thing I’ll say is that for anyone working in a Java EE environment where you have to use the stack that’s there, the future is bright and it’s all because of JRuby, Rails, and the speed and agility at which you can build applications on that framework. I’m convinced that JRuby is the best way to deploy a Rails app if you need performance and flexibility. My prediction: next year will be the year for JRuby’s rise into the mainstream.
53 comments