At random intervals throughout the day, we were receiving calls / emails from internal staff that our primary software portal was running sluggishly.  I first wrote it off as a latency issue, because the internet connectivity in our office is really bad, but it soon became apparent that customers outside of our network were seeing the problem as well.  I rebooted IIS and called it an evening, but my troubles were far from over.

Early the next day, my phone was buzzing with emails from my boss, telling me that our clients in Asia were complaining about site performance again.  I got my groggy butt out of bed, took at hot shower, and rode the train into work, ready to kick-butt and solve problems.

But, where to begin?

This is an asp.net application running on a windows production server, running on code compiled in “release” mode, on a machine floating around in the  Amazon EC2 ether.  I don’t have any stack traces, or other exception reports to work from.  I am effectively flying blind.  Where do I start.

Well, thankfully this was not my first Rodeo, and I have a few goto tools to work with:

Process explorer - How did I ever work without this amazing tool?  Gives solid metrics and other data about all processes running on a windows machine.  CPU utilization looked normal, but would spike to 100% and remain pegged for quite some time at random intervals.  Ram usage looked fine.  Next.

Process monitor – Basically tells you everything (and I mean everything) that every process doing at any given interval.  Really helpful for finding IO problems.  IO of all kinds is your biggest bottleneck in any software system, so being able to see if disks or network are being abused is key for flying blind.  Disk IO seemed fine, but network IO was through the roof.  Wireshark!

Wireshark - Due to my C++ background, I will often begin thinking of problems at the protocol level (I had to implement a SOAP webservice integration at the socket level in C++, but that is a story for a different day).  Wireshark is my goto.  A bit daunting at first, once you understand the lingo, it is invaluable.  As is often the case, wireshark had a pretty interesting story to tell.  During the CPU spikes, network traffic was through the roof.  At first, I thought that it might be a DOS or some other external attack, but upon further inspection, the vast majority of the traffic was between the EC2 instance and the RDS server (database).

Luckily, the traffic between IIS and the database is not encrypted, so I could see what sql statements were being run.  At first glance, it appeared normal, but once I was able to time a wireshark network traffic capture exactly during a CPU spike, which thankfully, happened by fluke chance due to the extremely sporadic nature of this problem, I saw some really strange SQL running over the wire.  About 98% of the statements were an insert statement into our analytic session table.

Session table?  There should only be one session row created per session upon successful login.  Strange.  After running a few queries, it turns out that there were a few THOUSAND session records created per login.  Eh…

Checking the session records, it turns out that the vast majority of the rows (about 80%) are duplicates and have the same timestamps (in groups of a few thousand).  Furthermore, those timestamps correlate to the exact times that we were seeing problems with performance.

Well at least I know what the symptom looks like.  What now?

I ended up writing a script that would run a database query looking for the duplicate (ghost) sessions and send me a text message.  That way I could login to the production server while the problem was happening.  It was actually pretty eerie being woken by a text message that said “Ghosts are here!!!”.  It always took me a few minutes of utter sleep deprived confusion to remember what the hell that actually meant.

After a few days of ghosts haunting me, I was finally able to isolate the problem down to a few lines of code:

Can you find the careerbug?

Well, I had the great fortune to be given free reign to build an entire software arm of a company from scratch.  Just think, carte blanche on server Infrastructure, application stack, ticket tracking system, source control type and hosting, release management, version management, development policies and practices, the “whole shootin’ match!”

Ok, well I guess that I inherited most of the application stack decisions of our previous contractors, but you get the point.

I felt like Augustus Gloop in the chocolate factory.

Server Infrastructure

One of the first tasks that I was given as the new lead developer, was to consolidate all of our existing server infrastructure.  We had 3 different service providers, and the costs were getting out of control.  I was told to pick a host, and consolidate.  Amazon Web Services (IAAS) to the rescue!  EC2 really is magic.  I say, “Amazon, I need 2 large linux servers running linux, 2 large servers running windows, a few mysql data stores, and a few mssql data stores”.  It thinks for a few minutes and says “Ok Jonfen, here you go!”)  Magic.  I have a few minor complaints with AWS, but overall, I have loved working on this great platform.

Development Machines

From the beginning, I knew that I had an interesting challenge ahead me.  I would be responsible for software components built in Rails, Python (django and cherrypy) and asp.net.  I wrestled with the idea of just using MacbookPros for both rails and asp.net development, but visual studio is already clunky enough without the virtualization overhead, and dual booting with base camp is tedious.  I instead opted for having windows development machines alongside our existing MacBooks (synergy anyone?)  This has worked out great so far, but I can’t tell you how many times I have tried and failed to copy-paste from my windows machine to my MacBook and vice versa.  windows (command) + c, ctrl + v?  Or windows + c, windows + v?  Buh…

SCM

I decided to use a mix of git and mercurial for our source control systems.  Previous contractors had used git on unfuddle to track our *nix projects, and I was fine with keeping that going (git that is, unfuddle is a cluster).  We use our MacBooks Pros for our *nix development, and I love how straight forward SourceTree makes working with git.  I know that the hardcore git guys will say that it is blasphemous to use anything other that the cli for git, but I think that source control should be a visual experience.  I migrated all of our existing applications from unfuddle to bitbucket git repositories.

The latest project that my team inherited was implemented in asp.net.  I tinkered with the idea of using git for this as well, but git is not pretty on windows (git bash?  Yikes…)  The DVCS of choice on windows seems to be mercurial.  The tortisehg client is mature and highly functional.  I had used it in conjunction with bitbucket on some personal projects, and I really liked it.

Git + SourceTree on OSX, Mercurial + TortiseHG on windows.  Cool.

Source Code Hosting

No contest here.  I needed a cheap hosting service that provided unlimited private repositories for both git and mercurial.  Whammo, bammo bitbucket!  Having been a happy free tier bitbucket user for a year or so now, I was excited to begin using it at work as well.  The closest competitor was github, and they aren’t even close (can you say pricey?)

Issue Tracking

I spent a while on this one.  Maybe its because there are more ticket tracking systems than Baskin and Robbins ice cream flavors.  I had worked with a wide enought variety of different systems at previous jobs (bugzilla, flyspray, helpdesk, team-forge, fogbugz) to make a good decision on what would be the best fit.  Issue tracking is the bread and butter of software development, so I wanted ot make sure to do this right with the following goals in mind:

• Must be a hosted solution.  I did not want to deal with managing the server on this.
• Must be simple to use (create tickets quickly and add as much detail as needed later)
• Must integrate with source control system to tie tickets to commits
• Must have good reporting
• Must have simple interface for tying tickets to release milestones (or sprints)
• Must have simple work-flow

Once I laid out all the criteria, this one ended up being a no-brainer as well.  Jira + Greenhopper.  Not only does Jira meet all of the requirements listed, it has seamless integration with bitbucket built in! (atlassian bought bitbucket a while back).  The greenhopper gives me a great view into each release and a quick glance view of everything that my team is working on.  I love it.  I honestly can’t picture working in any other environment after working like this.

Release / version management

After researching different methodologies for releasing, I decided that a two branch (default and release) approach would work the best for our needs.  All daily development is committed and pushed to the default (development) branch.  Once a release is ready, all commits are merged from the default branch into the release branch, and the release branch is tagged with the current version.  If a critical bug is found in the release, it is fixed, committed, and released directly from the release branch and merged into the development branch (export / import revision).

I like the idea of giving a particular version or sprint some personality, so I opened up to my team for ideas on naming our releases.  One of my engineers suggested naming releases after a particular genre – artist – album to match the major, minor, and tiny revision numbers.  It was determined that “metal” would be our first genre for “0″ series builds, so our first release was “0.1.1, Metal – Metallica – Metallica.”  While running our automated deployments, we play a song from the album for the release.  We have a great time with it.

Deployment

One thing that our previous contractors did very well is enforce push-button deployment for our software components.  Both ruby and python have great deployment frameworks (capistrano and fabric) that work over ssh.  I really like the simplicity and effectiveness of these systems.  There really is something so gratifying about typing “cap production deploy” and watching the magic happen.  After seeing the nightmare that deployments became at my previous job, I wanted to strictly enforce fully automated deployment.  Unfortunately, asp.net deployment is messy (integrated with visual studio or msi installers, uh no thanks.)  After considering a few commercially available systems, I decided to do what no programmer should do but always does and roll my own system.

I would structure it to work the same way as capistrano, but on windows.  Possible right?  You bet your boots!  In about a week of development time, I had our fully automated deployment system written from scratch in Python.  ”python deploy.py production”, cue the metal, BAAAM!  I love it.  I am so proud of my little push button deployment system that I am considering generalizing and cleaning it up a bit and posting it on github.  Fully automated asp.net server deployment anyone???

Anyway, I am very happy with how smoothly everything is running.  It is extremely gratifying to build an entire development stack from the ground up.  I hope that all you developers out there will get the pleasure one day.

In our solution, we are currently using the following technologies (in no particular order):

• Ruby / Rails
• Ruby / Sinatra / Unicorn
• MySql
• C# / ASP.net
• MSSQL
• Python / django
• Python / CherryPy
• Mongodb
• PHP

Quite a laundry list no?  This might sound like a crazy stack, but other software companies I work for are no different.  Software systems are complex beasts that tend to be spread across multiple platforms / technologies.  Why is this the case?  I believe that there are a few reasons why:

1. Software moves a million miles an hour – New programming languages, frameworks, and technologies come out seemingly every day.  Most of the time, these technologies provide developers with a faster, better, or more friendly way of solving a problem, so developers are anxious to get their hands dirty and try them out.
2. Programmers are magpies, and prototypes, unfortunately, often end up being end product – Because of reason #1, developers will often want to keep up on new technologies and frameworks by getting a “hello world” fired up.  I personally do this quite frequently.  It is critical for developers to stay current on new tech. Often while experimenting with new technology, a particular software problem will come up at work, and a developer will think “man, that would be soooo slick in Node!”  They poke around a bit, get a prototype running, show it to their customer, and before you know it, you’ve got Node running on your production server.
3. Technology choice, or lack of adapting new technology is a narrative of the culture of an organization – Organizations say a lot about who they are and want to be by what technology stack they use.  No one wants to be perceived as being built on a barn, so new passing craze technology will get wrapped into existing solutions just so the name of the technology can be thrown around in geeky conversation.

Seldom will a developer work for an organization that utilizes a single platform.  We need to be flexible.  We need to be willing to learn new things.  We need to be generalists.

Two stories:

At a previous job, a developer was working on a project.  He had worked in a particular technology for most of his career and was quite skilled at getting things done with it.  A request came in from management which required him to step outside of his sandbox and use a new technology.  I had personally used this technology only a little bit (due to my incessant “hello world” dablings) so he asked if I could help him out.  I of course was happy to help.  Teaching and mentoring are my true passions.  I began to explain to him how he could go about solving his problem with this new technology.  After about 5 minutes, he said something like this: “You know, why don’t you just go ahead and write the code for me.  I really don’t have any desire to learn new technology.”

I was floored.  How could you not have the desire to learn new technology?  How do you expect to stay even close to relevant as a developer?

The second story relates to my new position.  I am currently in the process of hiring a new developer, and I have had a few phone screenings with potential candidates.  One candidate in particular stuck out to me.  his resume was solid, and he had a lot of good .NET experience (which is good, because a large chunk of our new product is written in ASP.NET).  After speaking with him for a while, I asked him what he thought about Ruby, Python, and a few other technologies.  He said something very simmillar to the guy in my first story.  ”I haven’t worked at all in Python or Ruby.  I don’t like Linux.  I don’t really want to learn any of that stuff.”

He was immediately off my list.

A large chunk of our product is written in Ruby and Python, and runs on Ubuntu Servers.  What happens when those components break down?  Does he just twiddle his thumbs until a bug report for the ASP components come in?  Sorry.

Don’t be afraid to learn new things.  Don’t pigeon-hole yourself into a technology.  Be a generalist.

Unless, of course, you don’t want to be on my team.

Since I haven’t written anything earth-shattering, I most likely won’t be migrating over my old articles, but I think that I will keep my same bio and information.

Stay tuned.