The press release can be found here: “Sauce Labs Adds Windmill Test Framework Co-Creator Adam Christian to Engineering Team“.

Slide Inc.

I had a fantastic experience and learned a ton working with the really talented team of engineers, artists and product managers over at Slide. It was incredibly educational to work in an environment where so many people use your product everyday. I built a lot of really cool features there for SuperPocus and spent a year building a test automation infrastructure, and molding Windmill to be able to test anything and everything they put in front of me.

Slide was really amazingly understanding as I went through some personal struggles over the past few months. I will miss the people the most, as Slide created a great environment enabling people to effectively work together to solve really challenging problems.

Sauce Labs

Since I moved to the Bay Area, more than once I have seen people leave jobs to goto what they deemed their “dream job”. I never really understood what they meant by that designation until now.

Sauce Labs is solving exactly the class of problems that I find the most interesting, challenging, and sought after by so many people. I’m incredibly grateful and excited to be a part of the team working to make running your tests in the cloud seamless and fast, instead of frustrating and painful. The crew of people I will be working with are second to none and I look forward to learning all I can from them.

Sauce expects everyone to work directly with customers to ensure the best experience, and I look forwarding to helping many new teams get setup with test automation.

Future

It’s hard to outline exactly what future projects I will be involved in, as the technology is moving forward incredibly fast. I do know there are so many ways that we can make the testing community stronger, and the tools better. I see NodeJS and CouchDB opening many doors to new innovations and I would like to continue improving my Python skills.

I will still be within a block from South Park, so let me know if you are in the area and want to grab lunch!

Choosing Tools

A common first reaction is to take a couple of those old boxes sitting around to run the first “couple” tests you have. In some cases, this is the perfect solution. That is if you have a small application that rarely changes and only needs a daily test run (on one operating system and it’s available browsers). In my experience you can reasonably run one Windows VM without lagging the host machine unusable, which gives you two concurrent browser test jobs without worrying about process conflicts. Do remember that to do this correctly, you really want a machine dedicated to your CI system (which I will talk more about below).

VM’s are a great solution, however they require significant hardware overhead, continuous attention and licensing costs. Depending on the VM solution you choose you may also wind up dealing with the dreaded VM Time Drift causing problems with JavaScript and failing tests that aren’t actually failing.

“The Cloud”, is the 2009-2010 buzz word that makes all technology sound better, and in a lot of ways COULD be the ideal solution for test automation. You get the benefits of paying for only the cycles you use, having someone else manage the infrastructure and avoid those pesky licensing costs. However in my experience the setup is painful, the solutions (EC2, etc.) are slow and lacking some of the features to really do test automation well. For example, if you want to run your own CI instance and run your tests on demand in the cloud you will run into some pretty painful engineering problems. It’s not easy to instruct the “cloud” service to fire up a Windows VM and then have that VM connect to your CI instance and become an available slave. It’s doable with Linux, but last I checked – the features to do something similar with Windows simply weren’t there. Also do you really want to wait sometimes up to 15 minutes (also my experience with EC2) for the machines to come up before you can even start running your tests?

What CI System?

The point of this article is not to recommend solutions, it is to encourage questions. However, outlining all of the possible CI systems would take forever so I will simply say that I wound up using Hudson. The reasoning includes a very open and functional Open Source community, with smart contributors willing to take a few minutes to respond and help me out. I also found it possible (not simplistic) to build plugins to customize the things that I needed changed. Many people out there swear by Cruise Control, or Build Bot and I would highly encourage you to do some research and pick the solution that you feel will allow you to be the most productive. For example if you plan to use Windmill and EC2, you may want to do some reading about the Amazon EC2 and Windmill plugins available for Hudson and see what comparable tools are available.

Which test framework?

Some of you may know that I hold a mild bias when it comes to this question, but over the last year and a half I have ventured out into the land of testing frameworks and am able to see the values held by the other projects. For example, if you don’t ever need to deal with JavaScript in your application (or have browser specific functionality) — I suggest you use a tool that doesn’t require a real browser (like Twill). Tests will run faster, they will be more stable and can be run without access to your OS’s graphics layer.

When it comes to browser based web testing tools I really think you need to pick the one that fits your needs the best. A great example of this was in my needs to automate functionality contained in iframes being served over HTTPS from a different domain, the only solution (after weeks of trial and error) turned out to be WatiN. Of course, writing and building tests in C#.net wasn’t going to be an easy sell so IronWatin was invented as a means to write WatiN tests in Python.

Watir has captured a lot of the Ruby community and has recently been moving towards consolidating the separated browser  projects into one, which will significantly improve the ease of use.

Windmill has a dedicated community, focuses on dynamic JavaScript applications, boasts lots of features and goes for an ease out of the box type of experience.. at least that’s what I would like to think! Please feel free to check out the destination site or Github wiki for more information

Selenium has a thriving community, lots of available consulting support, integrates well into a Java environment and offers the Grid project. You can also avoid all of the work involved in running your own system by writing your tests in Selenium and then offloading them to a company like Sauce Labs if you are willing to pay for it.

What do we do about Flash/AS3 automation?

After unsuccessfully trying out FlexMonkey and AsUnit (don’t take my word for it), Matthew Eernisse sat down and wrote an AS3 test controller that works the way the rest of test automation works in the browser, it’s part of the Windmill codebase (codenamed FlashMill). There are two ways to run the tests, one is to hand FlashMill your tests already written in AS3 and the other is to write them in your favorite tool (or raw JavaScript) and have them call into the FlashMill API. Windmill currently has full IDE/UI integration in master to be released soon, the integration code is simple and can be viewed here (best doco at the moment until I write a better one).

Workload

Depending on the machine setup you are going with, this will vary. Obviously if you have a box and a VM on it you can manually go through the process of upgrading the browsers on each, installing patches and security fixes for the OS etc. But if you went with the VM solution, you need to come up with a way to deploy updates to all the machines in your pool. An Open Source solution that came up near the top of my search is WPKG, but like the rest of the tools on this page – there are many solutions and you will want to do your research. Some of the maintenance you will be dealing with can be done as a system job, or run by your CI system. A good example of this is to remove data, test files, source repositories etc. that accumulate on your test running machines.After a while, these files in combination with temporary internet files from the browsers and system tmp files start to slow things down.

This piece is very important to take into consideration from the beginning, because once you have 15 VM’s running tests — doing anything manually becomes a major chore. You also need to be cognizant of that fact that if you chose the cloud testing solution you will be managing your own test running images used to boot the VM’s. Every time you want to make updates or changes to that image, you get to go through the whole process of baking it and uploading it to the cloud hosting service. In my experience, this process is NOT enjoyable or quick.. so be prepared to invest some serious time.

What is the strategy for scaling and expanding?

Clearly this will be dictated by the rest of decisions you made, but I think when you get to this point, the idea of buying and manually setting up more and more physical boxes starts to break down. Buying more and more machines to sit there running tests simply seems like a bad use of resources (and desk space). VM’s allow you to quickly replicate images and expand your arsenal as long as the host machine has the hardware resources to power it without negatively effecting test run times on all the others. There are also solutions out there that allow you to boot and shutdown VM’s on the fly, which provides some interesting possibilities in juggling system resources.

I have found that since Windows is the common platform that can run all the browsers I care about, having a large pool of identical VM’s running all the time is an easy way to queue up 100′s of tests and get results in a reasonable amount of time. I do think that this is the aspect that cloud services start to become more appealing. The idea of spinning up more and more virtual machines in the cloud (with essentially endless capacity) makes the idea of scaling those tests considerably less terrifying. If you can get over the generally slow spin up times, and have come up with a strategy of dynamically harnessing and adding those machines to the pool – you may have it made!

What format/language is best for our tests?

At this point in web testing you simply need to decide what you care the most about doing. Is it manipulating the page? Or is it interacting with a database?, or a little of both? If you can get away with writing your tests fully in JavaScript, I would recommend it. At least in Windmill the JavaScript tests run exponentially faster than the Python or Ruby tests do. However the libraries for communicating with databases, email, system services etc, may make Python, Java, C#, etc your ideal solution. Each of the tools has their own ways of doing things, and to come up with the ideal language really depends upon your system, what your test developers are comfortable with, and what your application platform looks like.

What operating systems and browsers do we need?

The answer to this question should come from the metrics of your user base, and will also help you narrow down a testing framework and your test machines. Some of the available frameworks simply won’t run on Linux, or support IE6. Are all your users using Google Chrome? Then you should probably make sure the test framework you use has a Chrome launcher. Historically I have concentrated about 90% of the available VM resources on the most popular platform (usually Windows), with the most popular browsers (Firefox latest release, IE 7 or 8 depending on the users, Safari Windows latest release, and Chrome). This gives me a pool of work horse machines that can crank out tests representing the majority. The other 10% would be divided into the higher percentage minorities which probably includes a MacOSX slave running FF and Safari and a Linux machine running FF and Konqueror.

Methodology

I have found that having your own test infrastructure really makes this one easier. Since your resources will probably be on the same network as your codebase, you can easily access and run unit tests.. however as soon as you start sending your tests off to the cloud you are dealing with some security/privacy issues and engineering challenges. If inorder to unit test your code you need the entirety of your code base available, sending a copy of it off to your image on the cloud for every test (job, run or even change set) over and over could become a major strain on your system and simply sounds like a bad idea. If you are counting on unit tests you will at least want a machine on your network available as a slave from the CI system for those jobs.

How do we report results, and stay tuned into failures?

Most CI solutions have many ways they can be configured to alert you of a failure, from email, irc, jabber to a phone call you can usually find some solution that will get your attention. At this point in time the norm appears to be jUnit compatible results in XML. I’m not a huge fan, but the available tools for parsing and aggregating jUnit into something useful is very appealing. If you are okay with a true/false, that will always work out of the box, but you will need to be prepared to put in a little extra effort in both your test development and test job setup to generate the jUnit report files.

What role does automation need to play in our process?

For maximal results you need to make the leap where QA and Development both understand that a failing job in the CI system is a show stopper and must be investigated immediately. This way people honor the continual nature of continuous integration. It’s job is to catch problems shortly after they are broken, point you directly at them, and continue failing until you have solved the issue or fixed the test. If this isn’t the process you want, having continuous integration may not be the solution you are looking for.

Another effective strategy I have seen is to base release viability on the status of your CI system. Don’t let the product go out the door until the ‘thoroughly defined’ suite of functional and unit tests running in continuous integration are all running and passing. It is easy to push off the process of updating failing tests until “later”, but everyone is busy, and later is usually never.

What part of our application should we automate?

This really should be computed by time and resources, as much as I would like to say “everything” I am well aware of it’s low probability. Pick your application flows that make you money, or really mean a lot to your users likely experience. Ensure that your application loads, they can take the happy path, give you some money and then leave. At least this way you can sleep at night knowing that, they may not be able to change their profile information but they can still pay your salary.

Conclusion

I use the ‘configuration matrix’ option to build a matrix of browsers to run the tests, this way I can have one job that represents a test run on multiple boxes and multiple browsers on each box. Drilling down allows me to see the results for each of these test runs within the job. (Configuration Matrix is awesome btw, except for one really annoying bug for, which there is a reasonable workaround)

Having many build resources, sometimes I want to move a job from one machine to another in order to equalize load on the different machines or to have a job running in ff2 and ie6 instead of ff3 and ie7. Thus the need for a uniform build step that will run the same way regardless of the machine or installed browsers, became necessary. Additionally the commands for running tests got bigger and less manageable all the time, so the time for a clean user interface finally came. This way I can automatically append arguments like ‘exit’ to keep the build step interface simple and clean.

The other simplification is that in the Hudson configuration page for the Windmill Test plug-in you can tell it to automatically call the contrib ‘clean_run.py’ script with the correct arguments (assuming clean_run.py can be found in your path).

As I am relatively new to the Java world, I struggled through the development process but thanks to some reasonable plug-in documentation, responses from the hudson dev mailing list and some IRC conversations with Kohsuke (thanks so much) I finally have something that sufficiently addresses my needs.

Screenshots

Project

The code is all available on GitHub and I welcome any improvements or input from the community as I know that continuous integration is a very important piece of the utility provided by Windmill and this pieces should be as easy as possible to setup.

The combination of this plug-in, and others like the amazon s3 plug-in or the hudson VMware plugin bring us that much closer to seamlessly creating test resources, running our functional web tests across all of our supported platforms and then throwing it away until it’s again needed (or using the CLOUD).I have also done some integration with virtual box that I have found to be very successful.

Download

• Hudson Windmill Plugin 0.4

Please log bugs and let me know what you think! If you are an avid Java and or Hudson plug-in developer and you are interested in contributing, please jump right in!

First off, I have to say that for those of you who don’t know, PyCon is a community organized event, and amazingly well done. I was impressed by the design of the conference, the way they had four talks going on at once and they tried to keep them in a similar interest track. Every talk I attended was at least “good”, and many were “great”. You could really feel a community vibe, and for a conference that had 800+ attendees in the middle of a major recession they had every right to be excited.

There were two major themes content wise that really impressed me, the first was an amazing amount of web framework focus. Django obviously being the twinkle in the eye of the community, but there were smaller communities for each of the other projects, Pylons-Turbo Gears, web2py and lots of tools built on top of them. One that struck me with some major promise is the Pinax Project. Their goal is to make it so that I don’t ever have to deal with building user registration and in site messaging… and all the other features expected for any site that has social network functionality.

The other major theme was a mini testing conference going on within PyCon, that I was very comfortable hanging around with. We had a hugely successful Birds Of a Feather, as well as a surprisingly active Open Space talk for Windmill.

The “Using Windmill” talk turns out to be pretty successful in every aspect that I really care about. I do wish that I had been able to get a little more sleep the night before, and I have to admit the size of the venue was a bit overwhelming. I now realize watching the footage that I used the word “UM” way too much, and the demo videos must have been hard to watch from the very back of the room. But barring those two things, I am quite happy (I shall learn and practice for the next round of shameless PR at OSCON 2009).

The “Functional Testing Tools in Python” panel was very successful, and a lot of fun. I always enjoy the friendly banter between the different project owners. Everyone has a different opinion on what they care about, focus on and feel they do the best. Obviously since the only two projects represented that focused on Web Testing were Windmill and Selenium, we got a lot of attention.

Watching that panel footage I definitely think that the introductions were too long, but I still think our Journey themed – mind blowing – Windmill demo video was a great intro. At the very least, the audience had a little entertainment before the geek droning began

Slide had an awesome presence this year, a fun booth, huge banners everywhere and 6 attendees. It was fun to see all of the great responses I received about Slide from people out there in Python land.

Here are links to the videos:

• Using Windmill: http://pycon.blip.tv/file/1947236/
• Functional Testing Tools in Python Panel: http://pycon.blip.tv/file/1947342/

And some pictures:

If you are interested in seeing the new and improved version of the “Using Windmill” talk, please make it out to OSCON 2009, “Scheduled for 16:30 on 22 Jul 2009.” in San Jose, CA.

We are waiting to hear back from both Open Source Bridge and the AJAX Experience as to whether we will be participating in those conferences (fingers crossed)!

The pain began when I started in on moving our build slaves over to GIT. Of course two of the three are running Windows because we have to run the Windmill tests against IE. In an attempt to keep things simple, I wanted to avoid installing cygwin on the machines, so I tried msysGIT. Oh WAIT I have to get ssh to work before I can actually use GIT to pull down the repo. 

After trying and failing with openSSH, I finally realized that the Windows puTTY Package was the best way to go about this. There was much frustration involved with this process because it requires that you take the ssh key you generated on your mac (and had added on the server) and convert it to a puTTY ppk. Fortunately this turns out not to be that panful using puTTYgen. The next piece to the puzzle was to get pageant to load this key automatically when the machine boots. I went through some rigamarole trying to create a shortcut in the “Startup” items and appending a string to it’s path to get it to load the correct key. This didn’t work, I’m sure it was a combination of Windows being terribly un user friendly and my brain expecting things to “Just Work TM”. Finally I just created a short cut for the actual key and stuck it in the “Startup” folder, and it works! (You still have to enter your pass phrase on boot) but its better than the alternatives. I actually found a post on a puTTY forum where someone was asking to automate this piece too and the response was basically, “No, never, die. Doing that defeats the purpose of SSH”. 

One step that I forgot to mention is that to get GIT to use the right SSH key you have to set the environment variable: GIT_SSH=path/to/putty/plink.exe

Next I went and started my git clone, which kicked off perfectly. That wasn’t so bad! 46% later at 1.99GB of the retrieval process I received two fatal errors that looked similar to file system errors. I then did some Googling to find that msysGIT only supports repository’s up to 2GB. The answer on the forums for this problem was, “have a smaller repo”. This is a problem, so I head back to the GIT homepage, where I find that the only real lasting option is to use cygwin GIT. 

After I get cygwin GIT all installed I finally get the repo fully downloaded I check to see if I can use the msysGIT to pull changes (which should never be 2gb worth), and it absolutely freaks out. So that’s not an option, and neither is adding cygwin/bin to your path because that complains about ‘exec ssh’ not existing.

Your probably wondering, why don’t you just use cygwin to do your GIT stuff? Well the issue is that we are using hudson to queue up jobs on the machines using the cmd environment to pull changes and run the tests in the repo. And it became very clear that the best way to handle this was to run the slave agent from cygwin, that way the jobs are actually running in cygwin land and this turned out to work — awesome!

To finish describing my pain, the last problem was that since I am running the tests from c:\main, (you have to be inside a git repo to pull and I want to keep these jobs simple) I am generating the log file in the wrong place, because the job looks in it’s home directory to find the jUnit Compatible XML results file. Fortunately there are some environment variables that I can use to get the files back where I want; mv *.log %WORKSPACE%

Now that life is back to normal and I’m getting used to living in the new GIT world, I can get back to important things like making Windmill more awesome and increasing test coverage.

Cheers!

The first step is to get a couple machines that you want use as slaves and a machine to run Hudson, our setup looks like this:

Each of the machines with a different OS has Windmill installed. To make them slaves you simply bring up the Hudson web page on the machine, and run the launcher.. now it’s a slave — crazy easy right?

Now to setup test runs for the machines, in Hudson you click: “New Job” on the left hand side and do something like the following:

Tie this job to the slave you want it to run on (we can’t have IE runs happening on MacOSX):

Tell this job to run 10 and 30 minutes after the hour:

The build steps to actually run the tests, the first kills any straggling processes (more details below):

On the Mac for the Safari job, I want to make sure there aren’t any instances of Safari left hanging, or windmill processes sitting around so we do:
ps -ax | grep windmill | awk '{ print $1 }' | xargs kill | true
ps -ax | grep Safari | awk '{ print $1 }' | xargs kill | true


Then we want to grab the latest test code from svn and launch the windmill test:
svn up /Users/adam/Documents/main_bt/windmill/
python /usr/local/bin/windmill safari http://www.facebook.com test=/Users/adam/Documents/main_bt/windmill/fb email=username@slide.com password=pass report=true exit
rm /Users/adam/Library/Cookies/Cookies.plist


I am telling windmill to run a test against facebook.com, with the test hierarchy in the windmill/fb directory in Safari, with the provided email and password, then to report it’s results and exit.

The only thing different on our windows test runs is the way we kill the processes:
Example:
taskkill /F /T /IM windmill.exe
taskkill /F /T /IM firefox.exe


You might be asking how do I use those variables, check it out in my setup module:

1
2
3
4
5
6

def setup_module(module):
    client = WindmillTestClient(__name__)
    client.type(text=functest.registry['email'], id=u'email')
    client.type(text=functest.registry['password'], id=u'pass')
    client.click(id=u'doquicklogin')
    client.waits.forPageLoad(timeout=u'100000')

You can also read a great entry about adding reporting to your tests on Mikeal Rogers blog, here.

And that last line removing Cookies.plist makes sure that the next test run starts without any cookies set to cause problems.

Have Hudson keep you updated on Jabber:

Grab the generated XML output so you can view the test results in Hudson:

Do this for each of the test runs you would like to have, and boom — continuous integration:

This is obviously a simple scenario, and you can do way, way more customization.. but this should get you off the ground. Happy testing!