Category Archives: programming

Stack from scratch (almost)

4 Replies

What is your current technology / development stack?  Are you happy with it?  If you could start from scratch, how would you structure things?

Well, I had the great fortune to be given free reign to build an entire software arm of a company from scratch.  Just think, carte blanche on server Infrastructure, application stack, ticket tracking system, source control type and hosting, release management, version management, development policies and practices, the “whole shootin’ match!”

Ok, well I guess that I inherited most of the application stack decisions of our previous contractors, but you get the point.

 

I felt like Augustus Gloop in the chocolate factory.

 

Server Infrastructure

One of the first tasks that I was given as the new lead developer, was to consolidate all of our existing server infrastructure.  We had 3 different service providers, and the costs were getting out of control.  I was told to pick a host, and consolidate.  Amazon Web Services (IAAS) to the rescue!  EC2 really is magic.  I say, “Amazon, I need 2 large linux servers running linux, 2 large servers running windows, a few mysql data stores, and a few mssql data stores”.  It thinks for a few minutes and says “Ok Jonfen, here you go!”)  Magic.  I have a few minor complaints with AWS, but overall, I have loved working on this great platform.

 

Development Machines

From the beginning, I knew that I had an interesting challenge ahead me.  I would be responsible for software components built in Rails, Python (django and cherrypy) and asp.net.  I wrestled with the idea of just using MacbookPros for both rails and asp.net development, but visual studio is already clunky enough without the virtualization overhead, and dual booting with base camp is tedious.  I instead opted for having windows development machines alongside our existing MacBooks (synergy anyone?)  This has worked out great so far, but I can’t tell you how many times I have tried and failed to copy-paste from my windows machine to my MacBook and vice versa.  windows (command) + c, ctrl + v?  Or windows + c, windows + v?  Buh…

 

SCM

I decided to use a mix of git and mercurial for our source control systems.  Previous contractors had used git on unfuddle to track our *nix projects, and I was fine with keeping that going (git that is, unfuddle is a cluster).  We use our MacBooks Pros for our *nix development, and I love how straight forward SourceTree makes working with git.  I know that the hardcore git guys will say that it is blasphemous to use anything other that the cli for git, but I think that source control should be a visual experience.  I migrated all of our existing applications from unfuddle to bitbucket git repositories.

The latest project that my team inherited was implemented in asp.net.  I tinkered with the idea of using git for this as well, but git is not pretty on windows (git bash?  Yikes…)  The DVCS of choice on windows seems to be mercurial.  The tortisehg client is mature and highly functional.  I had used it in conjunction with bitbucket on some personal projects, and I really liked it.

Git + SourceTree on OSX, Mercurial + TortiseHG on windows.  Cool.

 

Source Code Hosting

No contest here.  I needed a cheap hosting service that provided unlimited private repositories for both git and mercurial.  Whammo, bammo bitbucket!  Having been a happy free tier bitbucket user for a year or so now, I was excited to begin using it at work as well.  The closest competitor was github, and they aren’t even close (can you say pricey?)

 

Issue Tracking

I spent a while on this one.  Maybe its because there are more ticket tracking systems than Baskin and Robbins ice cream flavors.  I had worked with a wide enought variety of different systems at previous jobs (bugzilla, flyspray, helpdesk, team-forge, fogbugz) to make a good decision on what would be the best fit.  Issue tracking is the bread and butter of software development, so I wanted ot make sure to do this right with the following goals in mind:

  • Must be a hosted solution.  I did not want to deal with managing the server on this.
  • Must be simple to use (create tickets quickly and add as much detail as needed later)
  • Must integrate with source control system to tie tickets to commits
  • Must have good reporting
  • Must have simple interface for tying tickets to release milestones (or sprints)
  • Must have simple work-flow

Once I laid out all the criteria, this one ended up being a no-brainer as well.  Jira + Greenhopper.  Not only does Jira meet all of the requirements listed, it has seamless integration with bitbucket built in! (atlassian bought bitbucket a while back).  The greenhopper gives me a great view into each release and a quick glance view of everything that my team is working on.  I love it.  I honestly can’t picture working in any other environment after working like this.

 

Release / version management

After researching different methodologies for releasing, I decided that a two branch (default and release) approach would work the best for our needs.  All daily development is committed and pushed to the default (development) branch.  Once a release is ready, all commits are merged from the default branch into the release branch, and the release branch is tagged with the current version.  If a critical bug is found in the release, it is fixed, committed, and released directly from the release branch and merged into the development branch (export / import revision).

I like the idea of giving a particular version or sprint some personality, so I opened up to my team for ideas on naming our releases.  One of my engineers suggested naming releases after a particular genre – artist – album to match the major, minor, and tiny revision numbers.  It was determined that “metal” would be our first genre for “0″ series builds, so our first release was “0.1.1, Metal – Metallica – Metallica.”  While running our automated deployments, we play a song from the album for the release.  We have a great time with it.

 

Deployment

One thing that our previous contractors did very well is enforce push-button deployment for our software components.  Both ruby and python have great deployment frameworks (capistrano and fabric) that work over ssh.  I really like the simplicity and effectiveness of these systems.  There really is something so gratifying about typing “cap production deploy” and watching the magic happen.  After seeing the nightmare that deployments became at my previous job, I wanted to strictly enforce fully automated deployment.  Unfortunately, asp.net deployment is messy (integrated with visual studio or msi installers, uh no thanks.)  After considering a few commercially available systems, I decided to do what no programmer should do but always does and roll my own system.

I would structure it to work the same way as capistrano, but on windows.  Possible right?  You bet your boots!  In about a week of development time, I had our fully automated deployment system written from scratch in Python.  ”python deploy.py production”, cue the metal, BAAAM!  I love it.  I am so proud of my little push button deployment system that I am considering generalizing and cleaning it up a bit and posting it on github.  Fully automated asp.net server deployment anyone???

Anyway, I am very happy with how smoothly everything is running.  It is extremely gratifying to build an entire development stack from the ground up.  I hope that all you developers out there will get the pleasure one day.

On being a generalist

Leave a reply

I have an interesting job.  I get to build and lead a small team of developers to create a fancy web-based content distribution / market analysis tool.  This will be the first time in the history of the company that a full time team has worked on the software.  Up until now, they have had a mix of contractors and outsourcing firms provide their software needs.  These are talented folks who have been able to come up with a great product, but as you can imagine, the system has become a bit fragmented.

In our solution, we are currently using the following technologies (in no particular order):

  • Ruby / Rails
  • Ruby / Sinatra / Unicorn
  • MySql
  • C# / ASP.net
  • MSSQL
  • Python / django
  • Python / CherryPy
  • Mongodb
  • PHP

Quite a laundry list no?  This might sound like a crazy stack, but other software companies I work for are no different.  Software systems are complex beasts that tend to be spread across multiple platforms / technologies.  Why is this the case?  I believe that there are a few reasons why:

  1. Software moves a million miles an hour – New programming languages, frameworks, and technologies come out seemingly every day.  Most of the time, these technologies provide developers with a faster, better, or more friendly way of solving a problem, so developers are anxious to get their hands dirty and try them out.
  2. Programmers are magpies, and prototypes, unfortunately, often end up being end product – Because of reason #1, developers will often want to keep up on new technologies and frameworks by getting a “hello world” fired up.  I personally do this quite frequently.  It is critical for developers to stay current on new tech. Often while experimenting with new technology, a particular software problem will come up at work, and a developer will think “man, that would be soooo slick in Node!”  They poke around a bit, get a prototype running, show it to their customer, and before you know it, you’ve got Node running on your production server.
  3. Technology choice, or lack of adapting new technology is a narrative of the culture of an organization – Organizations say a lot about who they are and want to be by what technology stack they use.  No one wants to be perceived as being built on a barn, so new passing craze technology will get wrapped into existing solutions just so the name of the technology can be thrown around in geeky conversation.

Seldom will a developer work for an organization that utilizes a single platform.  We need to be flexible.  We need to be willing to learn new things.  We need to be generalists.

Two stories:

At a previous job, a developer was working on a project.  He had worked in a particular technology for most of his career and was quite skilled at getting things done with it.  A request came in from management which required him to step outside of his sandbox and use a new technology.  I had personally used this technology only a little bit (due to my incessant “hello world” dablings) so he asked if I could help him out.  I of course was happy to help.  Teaching and mentoring are my true passions.  I began to explain to him how he could go about solving his problem with this new technology.  After about 5 minutes, he said something like this: “You know, why don’t you just go ahead and write the code for me.  I really don’t have any desire to learn new technology.”

I was floored.  How could you not have the desire to learn new technology?  How do you expect to stay even close to relevant as a developer?

The second story relates to my new position.  I am currently in the process of hiring a new developer, and I have had a few phone screenings with potential candidates.  One candidate in particular stuck out to me.  his resume was solid, and he had a lot of good .NET experience (which is good, because a large chunk of our new product is written in ASP.NET).  After speaking with him for a while, I asked him what he thought about Ruby, Python, and a few other technologies.  He said something very simmillar to the guy in my first story.  ”I haven’t worked at all in Python or Ruby.  I don’t like Linux.  I don’t really want to learn any of that stuff.”

He was immediately off my list.

A large chunk of our product is written in Ruby and Python, and runs on Ubuntu Servers.  What happens when those components break down?  Does he just twiddle his thumbs until a bug report for the ASP components come in?  Sorry.

Don’t be afraid to learn new things.  Don’t pigeon-hole yourself into a technology.  Be a generalist.

Unless, of course, you don’t want to be on my team.