08
Oct 13

The whole “hiring developers” thing

Approaching a year ago I was asked to step into the VP of Development job at Sauce Labs, which in retrospect is the first time I have ever had a job that was basically “full-time” management. I had convinced myself that I would somehow magically find enough time to also do a full-time developer job, that didn’t quite happen. Over the last 6 months I have spent 3/4 of my time concentrating on building out a development team. As it turns out, this process involves all the things; finding/hiring developers, organizing budget and roles, interfacing with recruiters, salary and stock concerns, and finally and most importantly – trying to wrap my head around what a world class development organization looks like from a cultural perspective.

Throughout this process I constantly read blogs, reddit, hackernews, etc on this topic, and have to say – although hiring developers is challenging, this whole “there are no good developers anywhere!!!” thing, is quite simply wrong. We had three really talented developers start today, and have hired and on-boarded five others over the last two months.

Developers want to work for companies that build things they care about, where they get to work on interesting and challenging problems. They also want to work for a company that embraces values they feel good about representing. We have documented our values, and take them very seriously.

Strategy

There is quite literally no magic here, we worked our asses off and bent over backwards to create opportunities that are individually exciting for each hire. I’m not saying we compensate abnormally high or anything — we just listen to what people want out of their future, and really think hard about their fit in the organization. Additionally, we are direct and honest about the opportunity and our expectations before we start getting serious about an offer.

Once we mutually feel like the fit is good, we look at our budget and future hiring requirements and make them the best offer that we realistically can. I don’t like the game where you offer someone less than you want to, to leave room for negotiating. I appreciated the offers I’ve had in my career where they said “here is the offer, it’s the best we can do, please consider it carefully”, and that is exactly how I like to do it. Avoiding attachment to outcome is an important part of this process, not every offer is going to be accepted, but either way, we feel good about having done the best we could.

I should mention that we have developers in Australia, Canada, Armenia and Poland, as well as H1Bs from Argentina, Germany and France. Not to mention contractors in Hungary, Switzerland and Australia. It’s challenging to build a core team in your office and efficiently acquire the talent and skills needed to get your projects done, and we consider that carefully for each hire. Like I said, get creative – the world gets smaller everyday, you don’t need to limit your talent pool to the bay area.

Recruiters

Once the word gets out that you are hiring, you may get firebombed by every recruiting agency on the map. I’ve found that when it comes to outside recruiters, less is more. There are (at least) two styles out there — one involves volume, assuming that the more resumes they send, the higher the chances are you will hire one of them. The other involves a thoughtful systematic search, where you only see a couple resumes here and there — but they are much higher quality. We have worked with two recruiters resembling the latter, and have had good success with both. I try to be a really accommodating person, but at the end of the day you have to be firm when it comes to low quality resumes — this is potentially one of the worst ways to waste your time.

Discovery

Of course your personal network, and that of your current development team is a fantastic way to find developers – but after a hire or two, you will probably burn that out. I suggest that you connect with your community. We use lots of Javascript and Python at Sauce Labs — and have spent a lot of time and energy supporting and contributing to those communities. This has paid off for us both with hiring and for product/uptake (given that we operate in the developer tools space). We have sponsored well over fifteen developer conferences this year, and have sent developers all over the world – not to mention the meetups we lead and support on a monthly basis.

If you can’t have real/engaging conversations with developers at technology conferences and meetups, then I suspect your hiring challenge goes beyond finding good developers.

Internal Resources

We recently hired a recruiting manager, and I would suggest that everyone do this sooner rather than later. Not to alleviate work load (even though that is awesome), but because of the impression instilled by quick, thoughtful, responsiveness. From scheduling interviews, answering questions, screening candidates, interfacing with outside recruiters – to a warm welcome and on- boarding package on the first day. You want people to feel like everything in your process that pertains to them is deliberate, and ultimately — it should be.

Patience

It took a couple months to get a steady stream of resumes coming in through the website, and posted on plenty of job boards. I suggest the air-raid, try all the things at once and then focus on the sources of the best resumes — but be patient, because developing a successful hiring and on-boarding process is not something to be done in a hurry.

Disclaimer

That’s all I have for now folks, thanks for reading — and remember, these are simply my observations over the last year. I am open to the idea that we might not have mastered this yet, especially given the relatively small number of new hires we have made compared to bigger companies in the space.

Did I mention that Sauce Labs is hiring? :)


29
Sep 13

Geddy.js Rocks

About 7 months ago, I decided I was going to try to build a small web app for doing group events. The concept is nothing revolutionary, but I had some differentiating ideas and have a need to be continuously exploring new tools. The first evolution was built using Express.js, CouchDB and a custom model layer.

To be honest, this is a really nice prototyping tool chain — I enjoyed getting to know CouchDB views a bit better, and the freedom of writing everything from scratch. I made a ton of progress over about 4 months and was excited about getting it online so that I could show it to folks. As soon as I had one user (in addition to me) logged in, I realized that I had no security,  my routes had become a disaster, and I had no standardization in my REST API. Getting this thing to a place where I felt comfortable with real users was going to be a mountain of work, due to the mess it had evolved into. When you are building something in the evenings and falling asleep, it’s hard to be thoughtful about small decisions you make. Over time these small decisions drift, and after being away from the code for 2 weeks — you have no idea where anything is, or what it does.

I took the app down, and decided to do some research — now that I had this new context. And for future reference, diving in and building something, is the best way to learn about what you need from your tools. I realized that I wanted an opinionated web framework, strong enough opinions that as my code base grew and became more complex – I knew what to expect from my abstractions.

To get to the point, I wound-up re-building my app in Geddy.js. It had been a long time since I had given Geddy a test run, I believe my last serious look was before @mde went to Yammer. I was really impressed at how the project had evolved, and grown up. Spending time in the Node community, I had heard many comments like “Geddy is the Rails of Node.js” – and having never used Rails, I wasn’t sure how to take that, but felt an implied negative implication. Well, now I suppose I have a better understanding of how startups all over the place have built and shipped apps like crazy with Rails. Either way, that seems like a pretty bullshit reason not to use a tool that makes you productive – so I’m glad I ignored it.

I got my app back to feature parity in about 4 weeks, working about an hour a night. the #geddy IRC room on Freenode was incredibly helpful, I even went over to the Yammer office after work one night to have @mde help me debug an issue that was blocking me. He simultaneously helped me move my whole app to using migrations. Geddy unfortunately doesn’t have CouchDB support (they do have Riak and an adapter could be built if someone took the time), but I moved over to Postgres. I have come to appreciate the confidence I have in making changes to my models, the performance, and how straight forward writing deploy scripts to the “production” instance are, using Postgres and Geddy models.

I’m gonna be honest, I don’t really care that much about technology religion or the purity of my “stack”. What I really care about, is the ability to put in minimal time, and build the product vision in my head, so that people can use it, before I get bored and move on to the next thing. At the end of the day, it’s really about the end result – and my plan is to deploy a small useful, tested, side project every couple months that my friends and family can play with. The current state of software development is a beautiful thing, take the massive set of available packages, and stitch them together into something cool and useful in a short amount of time.

I find the geddy command line tools really useful in my development workflow, which I call “a feature a night”. I log all my ideas in GH in bite size chunks, and before I go to bed each night — I try to hammer out one feature. ‘geddy scaffold newfeaturehere’ has become a part of workflow that I really enjoy. Geddy supports all the template languages (Jade, Swig, Handlebars, EJS, Mustache) – I picked EJS. Going forward would go with Swig (loved it as a Django guy and use it currently with my client side templates) – the markup just looks so much cleaner – and template languages should fail gracefully, right? It also supports all the DB’s I care about (other than CouchDB) – MongoDB, Postgres, Riak.

The most recent release (0.10.1) has fully featured migrations, customized error pages, environments (prod vs dev etc), workers (multiple processes handling load), and a CLI one liner will give you auth support using Passport.js.

The docs are good and getting better everyday, including step by step deployment directions of Heroku, Nodejitsu and Azure. And there is a whole section on internationalization that I haven’t yet explored – but plan to.

Before I go, I would like to underline – Geddy is full of suggestions on how to do things, but you don’t have to use all, or any of them. It really is just Node, straight forward, readable Javascript. And given the responsiveness from the Geddy community, if you have input, ideas, contributions — they will be encouraged and respected. As you can tell, I’m a pretty big fan.

Oh and if you wanna see the app I’m working on, it’s at http://www.activecove.com. I’m using Strider for Continuous Deployment, Sauce Labs for test automation, Rackspace for hosting, and Vagrant — but that’s another blog post.

GeddyCheckout the Geddy.js Project


08
Nov 11

The Perils of Language Learning

In 9th grade I had to pick between Spanish or French and even though I had French speaking family at home, I had absolutely no interest in either. And to make matters worse, I had a pretty serious condition called “computer programmer” – which made my 8:20 AM class about the last possible place I could ever learn anything. I chose French, and specifically remember waking up (having fallen asleep) watching some totally useless and uninteresting educational French video of two terrible actors standing in room talking about the appliances in their kitchen. I mean, what else are they going to talk about when it’s designed for 15 year olds? You can’t exactly talk about what France is actually good at — you know; food, wine and romance.

I endured this torture for two years, two different teachers and stacks of booklets trying to teach vocabulary – which were all lost on me. I do however specifically remember a good friend of mine picking the language up like crazy. She went on to take advanced French, visit the country with a class and eventually speak the language fluently. So I’m not blaming the system for my failures, it clearly works for some people. I’m actually quite jealous of people who took the opportunity to learn a language as a young person, because I’m convinced it’s significantly easier and much stickier in your brain.

A year ago September I had the opportunity to attend a technology conference in Berlin, and having never spent any time in Europe I thought to myself “When in Rome”. So I found myself some German language learning software and started at the very beginning. At this point, no one had told me that German is an obscenely hard language to learn. And after two months for an hour a day, I got on a plane and flew to Berlin. I immediately landed and jumped into a cab where I used the 5 German words I knew, and then listened to him speak in lightening speed German for the next 20 minutes of the ride (and I understood absolutely nothing). At that point it was painfully clear that in two months you can’t even learn enough of a language to properly talk to a foreign pet gerbil.

See languages are funny, they aren’t like anything else you learn – they are a magical combination of thinking and specifically not thinking – and that’s where the journey begins. My most important realization was when I realized that English was easy to me, because I never knew anything else. English is my Operating System, and I have been running it for 27 years – other people have been learning, evolving and using their language for years and years and years. How could I possibly be short sighted enough to think I could make any kind of usable progress in two months? I dunno, I’ve learned programming languages in less time.

Here’s how a language works:

Another way I like to envision it is that you start with this giant blurry photo of all the possible phrases, cultural understanding’s, vocabulary, and slowly overtime they become clearer and clearer until enough of them touch that you can identify pieces of the photo. The best thing about this metaphor is that when you look at the photo, you don’t think about each piece — your brain does all the heavily lifting automatically to help you identify the context of the photo.

As the human brain is so amazingly good at pattern recognition — you don’t actually require a super high percentage of clarity to know that this is a picture of a guy hiking in the woods next to a lake and a mountain.

So instead of giving up when I got back from Berlin, I kept at it — for another 11 months and as I built up more vocabulary — I tried as hard as possible to expose myself to new instances of the language. Each time I felt like I knew nothing, wrote down words and noted epiphanies. I must say, it really helps having people around who speak the language that you can ask questions. So much of a language is context and culture.

One of the most ground breaking parts of my learning experience was the discovery of music and television media that interested me in the language. (Go find yourself a native speaker and rack their brain). Starting with good music – because when you don’t even realize it’s happening, your brain is setting up patterns in your head that you will be shocked by later. Secondly, watching TV and movies really helps with an understanding of the flow of a language. As I found one of early, more difficult parts of learning was knowing when one word stopped and another began. We take our mother tongue for granted, we know where to expect pauses and have heard word combinations so many times that we know what to expect to come next.

Each word needs an enforcing moment, I’ve heard some people say – repeat it to yourself 3 times. Others say, correlate it with another concept as a mnemonic device. I actually found the best way to learn a word was to use it wrong, and get corrected. The simple moment of being embarrassed solidifies that word into your brain. And the only way for that to happen, is for you to go out there and try to speak to others that know the language. This is perhaps the most difficult part for me, as I prefer to practice presentations 5 times before I get on stage.

A helpful tip I found was to not attempt the translation of every word you hear, because you can never do that fast enough to understand a spoken paragraph (and it just makes you want to give up altogether). You have to just let the words pass through your brain, and over time you will realize that you understand more and more of what is being said. When you experience that for the first time, it’s pretty magnificent. I first noticed it after watching an episode of Family Guy – I walked into the kitchen pondering and forgot which language the episode had been in. I went back to the TV and found that the whole thing had been in German and inside my brain I had a full understanding of the episode void of any language attachment.

Don’t be afraid to fail, when kids first learn a language — they say things wrong constantly, and their parents constantly correct them. So why as we get older do we have such a hard time with being corrected? I see it in myself, but have grown to appreciate the effort in other people trying to help me out. This tip is specific to German, but don’t mix up “mich” and “mir” – man, they really don’t like that one. I suggest spending a couple extra weeks just working on using those two correctly. I specifically remember trying to practice my German one night here in San Francisco with a visitor from Munich, who politely, but firmly said “Please speak English to me, you may want to consider giving up on German”. And again, in Berlin where a German native said “Why would you want to learn German, you speak English right?”.

So this September a group of friends and I visited Munich and had a native German tour guide (now living in SF) to show us the Bavarian way. We spent some time at the Oktoberfest tents and touring the city – which was great fun. And I have to say, I’m quite happy that I didn’t give up because there were some moments of incredible satisfaction in my ability to communicate at least a little bit in another language. There was also a moment where I was faced with demonstrating my German language skills to a 6 year old that speaks 4 languages — I was terrified and my language skills reverted to the level of a kindergartner.

Some people would still say, “why learn another language?” – and I’m not going to try to convince anyone that it’s something they need to go out and do. However, I must say that it has exercised and affected my brain in ways that I don’t think I have ever experienced before. I have noticed that when I start to pay attention to a German song, or watch a German show that it literally feels like my brain is stretching. It’s an amazing sensation like nothing I have ever felt before, and recently for a few minutes everyday my brain has started to crave it. I think it’s easiest to compare to working out at the gym – it can be painful, but rewarding, and stretching afterwards “hurts so good”. And once you get into a routine – you start to feel a little out of sync when you don’t do it. I have noticed my memory has improved, and my problem solving abilities at work have received a boost. But the most interesting thing is that now I notice all kinds of things in the world around me that I never would have. A large percentage of the English language is heavily based on German, and I have a totally new understanding of why we say things the way we do.

I am currently taking a conversational German class at the Geothe Institute in San Francisco – and I must say it was a bit terrifying to walk in there with no academic backround in the language. But it was inexpensive, no pressure and has turned out to be a great experience.

Anyone with an interest in learning a language can do it – but I would urge you to seriously consider the size of the feat you are tackling. To really “learn” a language you have to make a serious time investment, but the more you put in – the more satisfying the experience can become.

I feel like I have just begun.

And remember.. some cultures are easier to get into than others. You can tell that I had a really rough time – but, “When in Rome”…


10
May 11

JSConf and NodeConf – 2011

Last week was both JSConf and NodeConf in Portland, OR and it was absolutely epic. I spend all year looking forward to the various Portland conferences, but this year happened to also be my first JSConf.us, the first ever NodeConf and a lot of thick bacon. If you aren’t aware of that term, it has become a great way to gauge the accumulated awesomeness regarding the quality of experience at a conference. I believe this was coined at one of the previous JSConfs and I have found it to be a useful addition to my vocabulary.

The opening party was populated with people dressed in amazingly authentic looking pirate costumes, many with a full size macaw parrot on their shoulder of differing and impressive colors. This party was sponsored by  Zappos, included a great pirate band, many fantastic local beers and they simultaneously ran a super fun twitter based prize contents. Towards the end of the night I won one of their awesome prize bags, which included an 8 bit green gameboy pocket (with tetris, super mario and metroid 2). This brilliant little old game device has been receiving a lot of attention at my house this week, somehow it is still incredibly addicting and fun to play. Way to go Zappos! I will continue buying nearly all my clothes through you.. :)

At 11am the next morning I had 45 minutes to announce and show off Jellyfish, which is a project I have been working on for the last few months. The goal of the project is to expose a simple API allowing you to run JavaScript in many different environments at once. In regards to my talk I did two stupid things: 1. I waited to record my demo’s until I arrived in Portland, 2. I waited to record my demo’s until after a beer tasting pirate party. Having said that, I think the talk went incredibly well and I have been really enjoying the responses and ideas I have received from the community.. keep it coming! I was happy to see a follow up talk filling in the bits my talk wasn’t able to. Matthew Eernissee talked about the pitfalls of writing JavaScript to be shared in multiple environments and gave Jellyfish a nice little plug.

I was thrilled to get a chance to hang out with some of the old OSAF gang, @towns, @twleung (who did two great writeups of JSConf and NodeConf), @mde, and @mikeal. It was interesting to hear all the different views on the attention explosion JavaScript has been experiencing over the last couple years, since we all worked on Cosmo — to this day Cosmo still has one of the most dynamic and interesting web based UI’s I can think of.

I’m glad to say that I was at the talk when Batman came to visit (batman.js) in order to help clear the conference of any potential criminals. I was also at the Yammer party with the great 80′s band where there was a Firefox running around. But the most interesting part of the whole experience was listening to the banter about where the future of the language should go. With Node.js doing it’s thing, people are using and seeing JavaScript in a totally different way. This has started to create different needs and expectations that didn’t really exist when JS development lived only on the client side. There are multiple different communities coming together to create this new “JavaScript Community” and watching them collide is sort of like witnessing plate tectonics from space. Each of the collisions creates a interesting new rift, or subduction zone in the mantel of the community.

One thing I can say for sure is that, you either get CoffeeScript, or you don’t — either way, I think it’s fair to say that we will continue seeing more and more ways for people to write JavaScript without writing Javascript.. and those people can get off my lawn.

If I had to pick a couple statements that sum up the crazy, amazing, drunken “JavaScript Community”, I would have to quote the two icons ultimately responsible for the happenings of last week. The first would be Bredan Eich explaining that “Javascript was thrown together to avoid other languages winding up in the browser” and that he’s aware that it isn’t an “ideal language syntactically”. The second being Ryan Dahl explaining that he’s not really a big fan of JavaScript, it is ” simply adequate for the need”, and he “hates callbacks”. Fortunately, both of these guys and the surrounding community leaders are all on the same page. Javascript is where it is because of it’s fortune of being “the browser platform”. We can all agree that as a language  there are a lot of improvements that can be made in harmony, I just hope that doesn’t mean that JS is going to turn into CoffeeScript. :)

The other unrelated thing I learned while at JSConf was that John Resig left Mozilla to build the community at the Khan Academy. I had never heard of the Khan Academy before, but I think it’s fair to say that I will probably re-learn everything I learned in college and more watching his videos over the next few years. Sounds like an awesome gig, and I wish him the best — I did find his Reddit AMA kind of amusing and totally bizarre – Mozilla is going to miss him.

I wish I had time to write an overview of all the JSConf talks, but to sum it all up quite crudely: Cloud9 and skywriter are the bomb, I think we will see more “learn how to program” tools in the browser with JS in the future (which is great) like waterbear. Also, apparently people care about putting JS on mobile devices — who knew?

As for NodeConf — this community knows that it needs to mature, and fast — so there was a huge emphasis on how to run big production systems on node, how to debug it and how to test it. I can’t wait to see the fallout from that day of collaboration.

I wanted to thank everyone for the great input on what I’m working on, and the ideas and feedback for Sauce Labs — I am taking all of it very seriously and am working on turning them into action items. For those of you two simply told me that you love Sauce and to keep doing what we are doing — I appreciate the boost!

Also,  I cannot forget a huge thanks to @voodootikigod, @mikeal and all the folks involved in running those conferences, I had a great time — thanks for inviting me to speak!

I took a lot of amazing photos, and I will be going through those this weekend – so stay tuned for a link to the photo album. And if you missed my Jellyfish talk, I will be doing it again at this years Open Source Bridge – June 21-24 back up in Portland, OR.

See you all at TXJS and JSEU, and maybe tonight at the San Francisco JavaScript Pub Night!


22
Dec 10

Node.JS is a Swimming Pool

Over the past six months I have heard about the unbelievable awesomeness involved in this project called Node.JS. At first I thought, “Server Side JS NOT on the JVM” – YAY! Then after a few days of that I thought, “I really don’t care about server side code, even if it is JS”.

See the thing is, right out of college I started fooling with raw DOM, cross browser event firing and capturing and building pretty big cool projects without the help and support of jQuery. So I developed this sick enjoyment of writing tons of raw uninhibited, IE 6 destroying JS. And ever since then, I have just thought of server side code as a necessary evil that I would PREFER someone else to write for me. Don’t even get me started on databases, I mean really — it’s hard to find software more out of touch with the web than traditional databases.

Anyways, so over and over I have pulled the git repo for Node, built it (a couple times had to Google error messages and change things to get it to succeed). Then I would get a ‘hello world’ working, and promptly close the terminal window and head back to my current Django based project (or close that terminal too and head back to client side JS). See, all along I just wanted to store my data on the client side and have some magic behind the scenes take that data and replicate it to a DB somewhere that I don’t need to know about. The point is that, Node wasn’t that magic and ‘hello world’ didn’t really feel that JavaScripty to me. I mean where is my ‘window’ object? And wtf is up with having to ‘require’ modules? I never got into client side ‘require’, so this concept is relatively foreign to me.

At JS.eu I started hacking on a node app, because I simply felt like a poser attending that conference and having not written much in Node. Honestly, minus a couple HTML 5 talks and some JavaScript icons like Crockford and Brendan Eich, JS.eu could easily have been called NodeConf.eu and no one would have really blinked. So I copied an Express app, got it to do some mundane simple tasks then tried to move to the next level. I got stuck due to some strange behavior that wound up being attributed to variable hoisting. At this point I put Node back in the drawer, see when I use a new technology I really want an impressive ‘hell world’ in less than 2 hours (without reading more than 20 minutes of docs). You may be thinking “wow that’s some serious impatience”, but I would be very surprised if I’m all that different than anyone else’s patience tolerance.

Finally about a month ago, I was about to dive into a new app using Django when @mikeal convinced me to use Node and put it on a no.de smart machine. After provisioning my no.de smart machine I was completely and utterly confused and frustrated. After a handful of questions @mikeal pointed me to: Countdown to Knockout: Post 11 – Deploying to Joyent which (although there have been some changes) made it clear enough that I could get my little app online and usable.

Once you have a working web app, the next evolution is to actually store data somewhere, and as I said before: “I am not a fan of traditional DB technology”. So naturally continuing the conversation landed me at Countdown to Knockout: Post 14 – Using CouchDB with node.js. In my ultimate laziness I couldn’t even be bothered to setup CouchDB on my Joyent smart machine, so I decided to use my CouchOne instance. Then I realized, the “CLOUD” buzz word, actually means; how many people can I pay to put my stuff on their machine? I wish I could also do my testing in the CLOUD, oh wait, you can, and I work there, see Sauce Labs.

So my main point is that for the last six months, Node.JS has been this very interesting swimming pool that I kept dipping my toes into — but it finally reached a reasonable temperature for me to dive in, swim around and discover a whole world (including mermaids like socket.io). Node isn’t awesome because it’s JavaScript, it’s awesome because it fits very nicely into this new development ecosystem where things feel like they are designed for the web. After I recognized the differences between Node and my client side JS habits, I was able to move back and forth between client and server side code without this painful mental context switch I normally experience  with syntax. It makes me hate server side development way less than I did in the past, and combined with CouchDB we find ourselves in a world where JSON is the native tongue.

Node may not be _the_ future of technology, but I sure see it playing an important role in my future, enabling faster application development. </rant>


15
Sep 10

CouchCamp was great

I just wanted to drop a quick note outlining some of my thoughts about CouchCamp (or you could call it “Swiss Family Robinson” meets database geeks anonymous).

First and foremost — having a tech conference at a summer camp is just plain awesome. As we all know that the social lubrication required to gel a bunch of geeks can lead to a state where said geeks should not operate any kind of a vehicle. In this case, all they had to do was walk 100 feet down a trail and pass out.

Secondly, I really like the idea that the beer follows you at a tech conference… I shouldn’t have to be running all over the place to find my beverage! Thumbs up to the CouchCamp staff for making that dream of mine a reality.

Did I learn anything? Actually, yes I did. I learned some really interested fundamental challenges that database ninjas face on a daily basis. I also saw a possible CouchDB roadmap being developed via the venting of current users, which was pretty neat.

As always Ted Leung came through to finish the grand event off with a great summary of the CouchDB community, where it’s succeeding and where it needs to go. I will never stop appreciating Ted’s open source jedi fu, it reminds me why I am here working in OSS.

It was great to meet everyone, see most of you at JS Conf EU!


03
Sep 10

September, Month of Travel

This summer has been relatively light on conference travel for me, minus some time in Portland for Open Source Bridge.. (which was awesome, again).

Next week I will be heading up to Couch Camp at Walker Creek Ranch. I am really looking for it for multiple reasons, but having visited the venue when it was being considered I couldn’t help but be blown away with the hidden little Marin valley. I really look forward to participating in the CouchDB geekery, general Open Source geekery, and outstanding scenery.

After which I have a week to prepare for a lengthy trip involving 360Flex, followed by a talk at the NYC Selenium Meetup, Sept 22 and then heading to JS Conf EU in Berlin.

While I’m on that side of the pond I am using the opportunity to go see some family in Lausanne, Switzerland and Aarhus, Denmark.

This will be pretty jam packed, but if you are interested in talking Open Source over a beer in any of these places.. let me know!

Also, in the spirit of all this I have updated my blog theme to what seems to be a lot simpler… maybe that will influence my state of mind, inspiring more frequent updates :)

One last note: Tomorrow the MooTools JavaScript Framework guys are hosting a hackathon at Cloudera, I am considering heading down for a bit in the morning to work on some cool ideas for a front end only Windmill test runner embed.


09
Jun 10

iPad’s, iPhone’s, iThink I’m going broke

Some of you may know that over the past few years I have been a bit of a early adopter, when it comes to Apple products. A few years ago, it was considerably easier — since the rate of product releases felt much slower and the increase in technological capabilities of each of the revisions seems significantly less compelling than today.

I remember thinking, I have a Macbook Pro and an iPod — what else could Apple possibly release that would be so compelling that I had to have it. As the upgrades happened, the iPod was still really just an iPod and the laptop’s got better but not that much better that you couldn’t wait two or three releases without feeling left behind. I feel that over the last year Apple has decided to call my bluff, and I have paid dearly.

As described by a “marketing person” a month or so ago, apparently I am a “card carrying geek”, and as a result cannot help but rally behind innovative technology. Growing up to Star Trek: The Next Generation, I am star struck by multi touch, video chat, GPS positioning, etc. thus I am the perfect target for these products.

I first have to say, I do this for all of you.. I mean, I can’t count the number of people who get in touch to ask about products before buying them — and I can’t let those folks down.

My little un-boxing ritual




I bought an iPad five hours after the release downtown San Francisco. I told myself, this time I’m going to be patient — I’m not going to join the insanity and stand in line like I did for the last two iPhone releases. And for five painful hours I sat at home watching the news and occasionally calling the Apple store to see how the stock was doing. Unlike my laptop and my iPhone, the iPad is not something that I need. In fact I’m having a hard time justifying it at all, considering the amount of devices sitting around my house.

HOWEVER, the iPad is a fantastic experience for the person who doesn’t have one of those two devices. It makes for a great browsing, emailing, casual gaming experience. Skype on wifi and 3g with iPhone headphones turns it into a totally legit way to make quick calls and the apps are getting better all the time. I personally am awaiting IOS4 to really make use of it. Pretty upset about the removal of the unlimited data plans by AT&T, if I was doing much travel or didn’t work in front of a computer all day, the iPad with unlimited data and Netflix would be an even more killer combo than it is on wifi.

My vision of the iPad is that it’s a start down a road to some exponentially cooler tech. The use case I invision is currently possible, but in a very early state. The single always online device you carry with your everywhere and dock in various locations for a more extensive experience. Many would argue that this same scenario applies to the iPhone. I still find the size of the iPad very curious, as it’s not small enough to carry in your pocket, but clearly too big to want to leave sitting in any one place.

I’m just going to come out and say this, I really like the look of the iPhone 3GS and preferred the weight of the original iPhone 2G. The new design doesn’t look very traditional apple to me, and so far looks significantly more invasive to my pocket. Fortunately the feature set is pretty amazing, but minus FaceTime and the gyroscope I’m wondering how much of the new experience I’m looking for will be achievable by upgrading my 3GS to the new software? I don’t feel all that compelled to do video editing on a tiny device, no matter how clear the resolution is. I understand that the speed updates are incredible, but will my 3GS experience start feeling unreasonably slow? I doubt it.

I just wanted to point all that out, so that after I try it out and wind up upgrading I have something to look back at with my starry eyes so I can answer all those questions (thus justifying the purchase).

It’s important to look back over the last few years and remember how the iPhone has changed things, I realize that is almost exactly what Steve said in the keynote. In Portland last week, I was walking home and my phone battery ran out after using the map all day long. For a few blocks I was pretty terrified because I realized that the first of my three reactions to my situation relied on, map, google, phone call. All the sudden my brain had to start doing things that it wasn’t used to doing! I’m not saying whether this is a good change or a bad change, I’m just saying it is a DRASTIC change.

Over the past month here in the bay area, plenty of people I know have been talking about how they *may* move over to one of the new android phones. They have these great big display’s with higher resolution photos, and it’s an open platform. I would be surprised if one of those folks actually made the change after this new phone becomes available… I’m just sayin.

It’s entirely possible that I’m an Apple fan boy living in the bubble, and recently I have been seriously thinking about running Linux on my MacBook instead of MacOSX (to mix things up a bit). But I would challenge all hardware makers to produce something, that head to head (in terms of user experience) can win me over.

So this is where I would normally try to conclude by casting my opinions as facts that you should really think hard about. Instead, I propose the question – without the innovation at Apple, where would the industry be? And for those of you who think I’m inside the bubble, enlighten me – what is going on out there?


23
Mar 10

My new gig – Sauce Labs

After almost two years of working at Slide Inc, I have started my new job at Sauce Labs.

The press release can be found here: “Sauce Labs Adds Windmill Test Framework Co-Creator Adam Christian to Engineering Team“.

Slide Inc.

I had a fantastic experience and learned a ton working with the really talented team of engineers, artists and product managers over at Slide. It was incredibly educational to work in an environment where so many people use your product everyday. I built a lot of really cool features there for SuperPocus and spent a year building a test automation infrastructure, and molding Windmill to be able to test anything and everything they put in front of me.

Slide was really amazingly understanding as I went through some personal struggles over the past few months. I will miss the people the most, as Slide created a great environment enabling people to effectively work together to solve really challenging problems.

Sauce Labs

Since I moved to the Bay Area, more than once I have seen people leave jobs to goto what they deemed their “dream job”. I never really understood what they meant by that designation until now.

Sauce Labs is solving exactly the class of problems that I find the most interesting, challenging, and sought after by so many people. I’m incredibly grateful and excited to be a part of the team working to make running your tests in the cloud seamless and fast, instead of frustrating and painful. The crew of people I will be working with are second to none and I look forward to learning all I can from them.

Sauce expects everyone to work directly with customers to ensure the best experience, and I look forwarding to helping many new teams get setup with test automation.

Future

It’s hard to outline exactly what future projects I will be involved in, as the technology is moving forward incredibly fast. I do know there are so many ways that we can make the testing community stronger, and the tools better. I see NodeJS and CouchDB opening many doors to new innovations and I would like to continue improving my Python skills.

I will still be within a block from South Park, so let me know if you are in the area and want to grab lunch!


02
Mar 10

Considering in-house web automation?

Recently I have had numerous conversations with people at various tiers of companies all over the place who are toying with the idea of building their own test automation and continuous integration infrastructure. Since I have spent a considerable amount of time dealing with such undertakings I decided that it might be worth the time to brain dump some of the issues you may want to consider before you dive in.

Choosing Tools


Boxes, VM’s or Cloud?

A common first reaction is to take a couple of those old boxes sitting around to run the first “couple” tests you have. In some cases, this is the perfect solution. That is if you have a small application that rarely changes and only needs a daily test run (on one operating system and it’s available browsers). In my experience you can reasonably run one Windows VM without lagging the host machine unusable, which gives you two concurrent browser test jobs without worrying about process conflicts. Do remember that to do this correctly, you really want a machine dedicated to your CI system (which I will talk more about below).

VM’s are a great solution, however they require significant hardware overhead, continuous attention and licensing costs. Depending on the VM solution you choose you may also wind up dealing with the dreaded VM Time Drift causing problems with JavaScript and failing tests that aren’t actually failing.

“The Cloud”, is the 2009-2010 buzz word that makes all technology sound better, and in a lot of ways COULD be the ideal solution for test automation. You get the benefits of paying for only the cycles you use, having someone else manage the infrastructure and avoid those pesky licensing costs. However in my experience the setup is painful, the solutions (EC2, etc.) are slow and lacking some of the features to really do test automation well. For example, if you want to run your own CI instance and run your tests on demand in the cloud you will run into some pretty painful engineering problems. It’s not easy to instruct the “cloud” service to fire up a Windows VM and then have that VM connect to your CI instance and become an available slave. It’s doable with Linux, but last I checked – the features to do something similar with Windows simply weren’t there. Also do you really want to wait sometimes up to 15 minutes (also my experience with EC2) for the machines to come up before you can even start running your tests?

What CI System?

The point of this article is not to recommend solutions, it is to encourage questions. However, outlining all of the possible CI systems would take forever so I will simply say that I wound up using Hudson. The reasoning includes a very open and functional Open Source community, with smart contributors willing to take a few minutes to respond and help me out. I also found it possible (not simplistic) to build plugins to customize the things that I needed changed. Many people out there swear by Cruise Control, or Build Bot and I would highly encourage you to do some research and pick the solution that you feel will allow you to be the most productive. For example if you plan to use Windmill and EC2, you may want to do some reading about the Amazon EC2 and Windmill plugins available for Hudson and see what comparable tools are available.

Which test framework?

Some of you may know that I hold a mild bias when it comes to this question, but over the last year and a half I have ventured out into the land of testing frameworks and am able to see the values held by the other projects. For example, if you don’t ever need to deal with JavaScript in your application (or have browser specific functionality) — I suggest you use a tool that doesn’t require a real browser (like Twill). Tests will run faster, they will be more stable and can be run without access to your OS’s graphics layer.

When it comes to browser based web testing tools I really think you need to pick the one that fits your needs the best. A great example of this was in my needs to automate functionality contained in iframes being served over HTTPS from a different domain, the only solution (after weeks of trial and error) turned out to be WatiN. Of course, writing and building tests in C#.net wasn’t going to be an easy sell so IronWatin was invented as a means to write WatiN tests in Python.

Watir has captured a lot of the Ruby community and has recently been moving towards consolidating the separated browser  projects into one, which will significantly improve the ease of use.

Windmill has a dedicated community, focuses on dynamic JavaScript applications, boasts lots of features and goes for an ease out of the box type of experience.. at least that’s what I would like to think! Please feel free to check out the destination site or Github wiki for more information

Selenium has a thriving community, lots of available consulting support, integrates well into a Java environment and offers the Grid project. You can also avoid all of the work involved in running your own system by writing your tests in Selenium and then offloading them to a company like Sauce Labs if you are willing to pay for it.

What do we do about Flash/AS3 automation?

After unsuccessfully trying out FlexMonkey and AsUnit (don’t take my word for it), Matthew Eernisse sat down and wrote an AS3 test controller that works the way the rest of test automation works in the browser, it’s part of the Windmill codebase (codenamed FlashMill). There are two ways to run the tests, one is to hand FlashMill your tests already written in AS3 and the other is to write them in your favorite tool (or raw JavaScript) and have them call into the FlashMill API. Windmill currently has full IDE/UI integration in master to be released soon, the integration code is simple and can be viewed here (best doco at the moment until I write a better one).

Workload


How much work goes into maintenance and software upgrades?

Depending on the machine setup you are going with, this will vary. Obviously if you have a box and a VM on it you can manually go through the process of upgrading the browsers on each, installing patches and security fixes for the OS etc. But if you went with the VM solution, you need to come up with a way to deploy updates to all the machines in your pool. An Open Source solution that came up near the top of my search is WPKG, but like the rest of the tools on this page – there are many solutions and you will want to do your research. Some of the maintenance you will be dealing with can be done as a system job, or run by your CI system. A good example of this is to remove data, test files, source repositories etc. that accumulate on your test running machines.After a while, these files in combination with temporary internet files from the browsers and system tmp files start to slow things down.

This piece is very important to take into consideration from the beginning, because once you have 15 VM’s running tests — doing anything manually becomes a major chore. You also need to be cognizant of that fact that if you chose the cloud testing solution you will be managing your own test running images used to boot the VM’s. Every time you want to make updates or changes to that image, you get to go through the whole process of baking it and uploading it to the cloud hosting service. In my experience, this process is NOT enjoyable or quick.. so be prepared to invest some serious time.

What is the strategy for scaling and expanding?

Clearly this will be dictated by the rest of decisions you made, but I think when you get to this point, the idea of buying and manually setting up more and more physical boxes starts to break down. Buying more and more machines to sit there running tests simply seems like a bad use of resources (and desk space). VM’s allow you to quickly replicate images and expand your arsenal as long as the host machine has the hardware resources to power it without negatively effecting test run times on all the others. There are also solutions out there that allow you to boot and shutdown VM’s on the fly, which provides some interesting possibilities in juggling system resources.

I have found that since Windows is the common platform that can run all the browsers I care about, having a large pool of identical VM’s running all the time is an easy way to queue up 100′s of tests and get results in a reasonable amount of time. I do think that this is the aspect that cloud services start to become more appealing. The idea of spinning up more and more virtual machines in the cloud (with essentially endless capacity) makes the idea of scaling those tests considerably less terrifying. If you can get over the generally slow spin up times, and have come up with a strategy of dynamically harnessing and adding those machines to the pool – you may have it made!

What format/language is best for our tests?

At this point in web testing you simply need to decide what you care the most about doing. Is it manipulating the page? Or is it interacting with a database?, or a little of both? If you can get away with writing your tests fully in JavaScript, I would recommend it. At least in Windmill the JavaScript tests run exponentially faster than the Python or Ruby tests do. However the libraries for communicating with databases, email, system services etc, may make Python, Java, C#, etc your ideal solution. Each of the tools has their own ways of doing things, and to come up with the ideal language really depends upon your system, what your test developers are comfortable with, and what your application platform looks like.

What operating systems and browsers do we need?

The answer to this question should come from the metrics of your user base, and will also help you narrow down a testing framework and your test machines. Some of the available frameworks simply won’t run on Linux, or support IE6. Are all your users using Google Chrome? Then you should probably make sure the test framework you use has a Chrome launcher. Historically I have concentrated about 90% of the available VM resources on the most popular platform (usually Windows), with the most popular browsers (Firefox latest release, IE 7 or 8 depending on the users, Safari Windows latest release, and Chrome). This gives me a pool of work horse machines that can crank out tests representing the majority. The other 10% would be divided into the higher percentage minorities which probably includes a MacOSX slave running FF and Safari and a Linux machine running FF and Konqueror.

Methodology


Can we support both functional and unit tests?

I have found that having your own test infrastructure really makes this one easier. Since your resources will probably be on the same network as your codebase, you can easily access and run unit tests.. however as soon as you start sending your tests off to the cloud you are dealing with some security/privacy issues and engineering challenges. If inorder to unit test your code you need the entirety of your code base available, sending a copy of it off to your image on the cloud for every test (job, run or even change set) over and over could become a major strain on your system and simply sounds like a bad idea. If you are counting on unit tests you will at least want a machine on your network available as a slave from the CI system for those jobs.

How do we report results, and stay tuned into failures?

Most CI solutions have many ways they can be configured to alert you of a failure, from email, irc, jabber to a phone call you can usually find some solution that will get your attention. At this point in time the norm appears to be jUnit compatible results in XML. I’m not a huge fan, but the available tools for parsing and aggregating jUnit into something useful is very appealing. If you are okay with a true/false, that will always work out of the box, but you will need to be prepared to put in a little extra effort in both your test development and test job setup to generate the jUnit report files.

What role does automation need to play in our process?

For maximal results you need to make the leap where QA and Development both understand that a failing job in the CI system is a show stopper and must be investigated immediately. This way people honor the continual nature of continuous integration. It’s job is to catch problems shortly after they are broken, point you directly at them, and continue failing until you have solved the issue or fixed the test. If this isn’t the process you want, having continuous integration may not be the solution you are looking for.

Another effective strategy I have seen is to base release viability on the status of your CI system. Don’t let the product go out the door until the ‘thoroughly defined’ suite of functional and unit tests running in continuous integration are all running and passing. It is easy to push off the process of updating failing tests until “later”, but everyone is busy, and later is usually never.

What part of our application should we automate?

This really should be computed by time and resources, as much as I would like to say “everything” I am well aware of it’s low probability. Pick your application flows that make you money, or really mean a lot to your users likely experience. Ensure that your application loads, they can take the happy path, give you some money and then leave. At least this way you can sleep at night knowing that, they may not be able to change their profile information but they can still pay your salary.

Conclusion


I hope that I sufficiently communicated my point and passed on some useful and informative tidbits. Setting up automation infrastructure that has any chance of actually doing it’s job is a major investment of resources and should be well planned based on individualized needs. I hope this saves someone some time and energy.

Best of luck, and happy testing.


Get Adobe Flash player