What I learnt at University

Posted by Annie on Dec 6, 2011 in Opinion, Technology | 4 comments

I had a great time at uni. I spent 4 years working my ass off trying to do as well as I could in my sometimes-impossibly-hard Comp Sci and Maths courses. Sure, I also met a lot of fun people, did a lot of fun stuff, held down a part-time Comp Sci tutor job and all that other stuff, but I studied really hard too. Most of my papers had weekly assignments hard enough to make most people want to quit and a lot of my papers weren’t worth many credits. I don’t know, sometimes it felt like my courses were harder and more work than some of my other fellow students.

Great, but was it relevant?

After being a Software Engineer in several different roles in the IT industry for 8 years now, I have come to the conclusion that I’ve never used (and probably never will) most of what I learnt at University. Academia is like that – you learn crazy hard stuff for the sake of learning. I’m not complaining though – I really enjoyed all the theory I learnt, I’m just not sure how useful / relevant it is in today’s typical software development roles.

Anyway, I’ve still got a few of my assignments from one of the hardest papers I did – COMP473 – Formal Aspects of Concurrent Systems. I’m not even sure they’re still running this paper at Victoria University of Wellington anymore, but here they are. I seriously haven’t done anything like this in any of my roles in the last 8 years:

• COMP473 – Assignment 1
• COMP473 – Assignment 2
• COMP473 – Assignment 3
• COMP473 – Assignment 4
• COMP473 – Assignment 5
• COMP473 – Assignment 6
• COMP473 – Assignment 7

Sadly, nowadays, I’m not sure I understand a word of what my answers actually meant! I can’t help but think that I’m getting dumber… but then I realize that most of the work I do these days isn’t rocket science and I’m just out of practice at solving complex problems. Perhaps I should take a new University course?

Software deployment processes

Posted by Annie on Oct 28, 2011 in Coding, Technology | 7 comments

During my career as a Software Engineer, I have seen a variety of different approaches to ‘deployments’. This is one aspect of the software development life cycle that wasn’t touched on at all during my 4 year Computer Science degree and seems to be a point of contention for a lot of software companies.

In my experience, a lot companies seem to almost fear deployments, leading to unmanageable 6-week deployment cycles, middle-of-the-night deployments that take hours and inevitably result in unplanned outages, a lot of error-prone manual configuration, unnecessary friction between developers and sys admins, no rollbackability, no version history, etc etc.

Image sourced from http://sgale.blogspot.com

Deployments shouldn’t be that hard

Really, they shouldn’t. If you have a well-planned, stable and automated deployment process, you should be able to deploy early and often and hopefully without any glitches. Your customers will be happier and you won’t end up looking like the poor cat in the picture above.

Below is a list of what I consider to be the most important qualities of a good deployment strategy (not necessarily in order of priority):

• Manageability
  Deployments really shouldn’t be as complicated and scary as a lot of companies make them out to be. It should be a one-button action and it shouldn’t require taking the whole system offline for hours in the middle of the night by the whole sys admin team. The number of steps involved in deploying may vary depending on your system and how major the deployment is, but they should be easy enough to follow.
• Configurability
  It should be fairly easy to deploy any of your projects – from static files to binaries, web applications that need IIS to be recycled and even windows services that need to be uninstalled, installed then restarted.
• Visibility
  Developers, sys admins and anyone else in the company who wants to know should be to quickly and easily be able to see a history of all deployments, what the last version deployed was, what the changes were, who worked on it and when exactly it went live.
• Rollbackability
  For those odd occasions when a production release doesn’t go quite as expected, it should be relatively easy to rollback to a previous version. I say relatively easy because it really depends on how major the deployment was and how complicated your software is.

My deployment system

In one of my previous roles, I was asked to come up with a process for deploying the various components that made up the software we were building. The components that needed to be deployed (sometimes together, sometimes independently) were .NET web services, ASP.NET MVC web apps, windows services and static files. What I ended up building was a deployment process that involved Teamcity, MSBuild scripts, custom MSBuild tasks, a .NET ‘deployment web service’ and last but not least, a page on our intranet to show version history and comments.

Since building this process, I’ve been asked to describe it several times so I thought to myself, why not blog about it?!

Requirements:
I built this system for one company, with their requirements in mind. It may not suit everyone’s deployment needs. For example, my deployment system did not deploy SQL scripts as this was not a requirement. Any scripts / tables that needed to be created or altered / migration scripts would be run manually before deploying the code. This became a part of the greater deployment process, but it was not handled automatically. And finally, I only built the code aspect of this system and wired it all up through Teamcity – our sys admins set up all required VPN tunnels, FTP servers, etc etc…

Process:
For every code solution, we had an MSBuild script that Teamcity would use to build and publish the binaries / static files on dedicated build agents. Apart from giving Teamcity instructions on how to build the solutions, these MSBuild scripts would also zip up the binaries and static files using the version number as the filename and then copy the zip files to a build server. The sys admins set up an FTP server pointing to all these zip files so that the servers we deployed to had access to them.

The next step was to set up new Teamcity build tasks that used different MSBuild scripts to actually deploy the zipped up binaries and static files. We had one MSBuild task per solution and environment. These MSBuild scripts used custom MSBuild tasks to call the deployment web service on a particular server / environment. In cases where a particular solution was load balanced, we would pass the host names of each server hosting the solution to the custom MSBuild task which would then spin up a bunch of threads and call the deployment web service on each of those servers.

The Deployment Web Service – the brains of the system:
The ‘deployment web service’ was the brains of the system – the rest of the stuff I just described simply wired everything up. The first step in setting up a new server was always to deploy and configure the deployment web service. Once it was installed and configured properly, this is more or less what the deployment web service would do when it was called:

• Download the appropriate zip file containing the built solution from the FTP server onto the server you are deploying to.
• Unzip the zip file into a temporary folder.
• Modify the config files – depending on the environment being deployed to, the deployment web service would pick the right config file, delete all the others and rename the remaining one to Web.config (or App.config…).
• If the solution being deployed was a windows service, the deployment service would now stop and uninstall it if it were already installed and running.
• Copy all the files from the temporary directory over the top of the actual binaries using robocopy.
• If the solution being deployed was a website or a web service, the deployment service would now recycle the app pool that the site was running under.
• If the solution being deployed was a windows service, the deployment service would now install it.
• Delete all temporary files.

Simple, right? Seriously though, that was the brains of the system and it really didn’t do anything out-of-this-world. Deploying binaries and/or files really isn’t that hard!

But what about environment-specific configuration?

This is a much debated point and I’m not claiming that the way we dealt with environment-specific configuration is the only way to go, but it was certainly simple and enabled us to have automated stress-free deployments.

For every application we built, we would maintain separate config files for each environment. That’s right – we had a Web.config (for localhost), Web.Development.config, Web.Test.config, Web.Staging.config and Web.Production.config. I know some of you reading this will be rolling your eyes right about now and thinking… “So every time I need to add a new key to my config file, I have to add it to 5 different files?!? What a nightmare!!”. Trust me, it really isn’t that hard to keep all these files in sync and seriously, it’s much easier than trying to manually merge config files at the last minute before deploying. Think about it this way – once a system is up and running, you don’t add keys anywhere near as often as you have to deploy (x number of environments you’re deploying to).

So how did the right config file get deployed to each environment? Well, all these environment-specific config files would be included as part of the project to be deployed. Each instance of the deployment web service knew which environment it was sitting in and deploying to (this was configured via it’s own web.config) so as I described above, the deployment web service would pick the correct config file based on it’s filename, delete all the others and rename the correct one to Web.config (or App.config). Easy peasy.

Benefits

Having used several different techniques for deploying code through environments and out to production over the years, this system certainly demonstrated various benefits and in particular, satisfied all of the qualities held by a good deployment process which I listed above.

• Manageability
  Since this system was triggered from Teamcity, it really couldn’t get any easier to manage deployments. Deployments became a matter of pushing one button and waiting for a couple of minutes to see the result. There was no manual merging of config files, no remoting onto the server to restart an app pool. Just one button click.
• Configurability
  In terms of set up, although this system is made up of several components, it was fairly easy to configure a new project to be deployed. Sure, there were a few steps involved but they were all simple and more or less a copy-and-paste of an existing project’s deployment scripts. It was also pretty straight-forward to set up a new server to deploy to. This involved setting up the deployment web service on the new server and updating any MSBuild scripts with the hostname of the new server for projects that you want to deploy to it.
  Because it was highly configurable, the system was able to deploy several components at the same time, to various different servers.
• Visibility
  Because we used Teamcity to trigger the deployments, we had all the whole build log that Teamcity produces. I put some logging into the custom MSBuild tasks so that we could see what was going on from within Teamcity’s build logs. Later on, I added some code to update a version databases so that deployments were visible from within our intranet.
• Rollbackability
  Rolling back to a previous version was a matter of looking up which version number you wish to roll back to and setting this version number in the parameters passed to the deployment MSBuild script by Teamcity. The custom MSBuild tasks and deployment web service would then pick up that particular version and re-deploy it.

Final thoughts

In over a year of daily deployments of several different web sites, web services, windows services and static files, my deployment system never failed. Having this system in place enabled us to focus on writing cool software and getting it out often and early instead of planning stressful deployments. Everyone benefited from this system – developers and testers could easily and quickly deploy to dev, test and staging as often as they wanted, without waiting for sys admins to be free, and production deployments were no harder. Sys admins didn’t need to know anything about how to deploy, managers were able to see when a new version had been deployed to a particular environment and all the automation meant that there was very little room for human error. A definite win for everyone!

If your team doesn’t have an automated deployment system already in place, I implore you to either build your own (like we did) or use an off-the-shelf one. It may take a bit of effort to set it all up at the beginning, but it’s definitely worth it in the long run.

Deploying ASP.NET MVC 3 to IIS 6

Posted by Annie on Oct 19, 2011 in ASP.NET MVC, Coding, Technology | 4 comments

I recently built a new ASP.NET MVC 3 (.NET 4) web application which unfortunately has be hosted on Windows Server 2003 running IIS 6. Guess I’d figured getting it running on IIS 6 would be as easy as installing .NET 4 and ASP.NET MVC 3 on the server… but it wasn’t. This isn’t a step by step tutorial on how to get this working, but more of a list of things to look out for and some links to helpful material which all helped getting my site up and running.

Installations

• Install the .NET 4 Framework.
• Install ASP.NET MVC 3. Apparently you can avoid installing ASP.NET MVC 3 on the server and just deploy all the appropriate MVC assemblies in the bin folder of your app instead, but I couldn’t get this to work.

IIS Configuration

• Make sure ASP.NET v4.0.30319 is ‘Allowed’ under Web Service Extensions in IIS – it is ‘Prohibited’ by default.
• Create a new app pool for your web site to use – IIS 6 app pools cannot run sites under different frameworks so make sure this app pool is only used for your new site.

IIS Website Configuration

• Under the ‘ASP.NET’ tab, make sure that the ASP.NET version is set to ‘4.0.30319’.
• Under the ‘Home Directory’ tab, make sure that ‘Execute permissions’ is set to ‘Scripts only’ (at least).
• Click on the ‘Configuration’ button in the ‘Home Directory’ tab and make sure that all the Application extensions are mapped to .NET 4 versions of the dlls.
• Add a ‘Wildcard application mapping’ – click on ‘Insert’ and enter ‘C:\WINDOWS\Microsoft.NET\Framework\v4.0.30319\aspnet_isapi.dll’ as the ‘Executable’. Leave ‘Verify that file exists’ unchecked – checking this made my routes stop working.

Permissions

If you’re running under an account with limited permissions, you may need to grant it ‘List Folder Contents’ and ‘Read’ permissions on the Windows Temp directory. I needed to do this because I’m calling out to web services and .NET creates temporary classes in the Windows Temp directory with which to serialize objects from XML. Not sure if this is needed if you’re not doing any serializations, but worth mentioning.

Conclusion

I kept getting 403 and 404 errors when I first deployed my new ASP.NET MVC 3 (.NET 4) web application to IIS 6, but after doing / checking all of the above, it all seems to be working smoothly.

And last but not least: if you’re allowed to, try recycling your application pool and resetting IIS if some changes don’t seem to be making much difference.

Useful links

http://haacked.com/archive/2010/12/22/asp-net-mvc-3-extensionless-urls-on-iis-6.aspx
http://johan.driessen.se/posts/getting-an-asp.net-4-application-to-work-on-iis6
http://blog.stevensanderson.com/2008/07/04/options-for-deploying-aspnet-mvc-to-iis-6/

CLR Versioning and Side-by-Side Execution – .NET 2.0 to .NET 3.5 and .NET 3.5 to .NET 4.0

Posted by Annie on Sep 16, 2011 in Coding, Technology | 0 comments

The other day at work I spent some time investigating an OutOfMemoryException being thrown by third-party component being used in a .NET 3.5 application. After a closer look and weighing up some options on how to fix it, we realized that the problem went away if the application was recompiled in .NET 4. This ended up causing some discussion among the team – how could simply recompiling the application that the .NET 3.5 third-party library was being used in fix the problem in the library itself?

One of our team members was convinced that if the third-party library had been built using .NET 3.5 and it was then used in an application that targeted .NET 4, that the third-party library would continue to run under the context of .NET 3.5. To that end, he was sure that recompiling the application in .NET 4 couldn’t have actually solved the problem.

Of course this triggered a Google search frenzy which lead us to this article on msdn:

Excerpt #1:

Add-in Problems:

To give all add-ins the best chance of working, we always choose the latest runtime for managed COM activation. Even if you only have older add-ins installed, there is no way for us to know that when that add-in gets activated, so the latest runtime still gets loaded.

An unfortunate side effect of this activation policy is that when a user installs a new application with a new version of the runtime, completely unrelated applications that use managed COM add-ins, built against older versions, suddenly start running on a newer runtime and can fail.

For the .NET Framework 3.0 and 3.5, we solved this problem through an extremely strict policy: each release was additive and only added new 
assemblies to the prior version with the same runtime underneath. This prevented any compatibility issues when installing them on a machine running the .NET Framework 2.0.

This means that when you are running an app on the .NET Framework 3.5, you are really running it on the 2.0 runtime, with a few extra assemblies on top of it.

What the above implies is that the update from .NET 2.0 to .NET 3.5 was only ‘additive’, that is, Microsoft added some new functionality (dlls) but kept existing functionality the same for backwards compatibility. Actually, if you take a look in the v3.5 folder under Microsoft.NET in the Windows directory, you’ll see that there are a handful of new dlls but no new core libraries to replace the .NET 2.0 versions.

Therefore, if the application we were dealing with had been using a .NET 2.0 third-party component and we were rebuilding the whole application in .NET 3.5, all core types would still be exactly the same under the covers and our colleague would have indeed been correct in his conclusion that rebuilding the application couldn’t have solved the problem.

However, this doesn’t seem to be the case from .NET 3.5 to .NET 4… if one reads on:

Excerpt #2:

Add-in Problems:

This means that when you are running an app on the .NET Framework 3.5, you are really running it on the 2.0 runtime, with a few extra assemblies on top of it. However, it also means that we couldn’t innovate in the .NET 2.0 assemblies, which include key functionalities, such as the garbage collector, just in time (JIT) and base class libraries.

With the .NET Framework 4 we have implemented an approach that allows high compatibility, including never breaking existing add-ins, and also lets us innovate in the core. We can now run both .NET 2.0 and .NET 4 add-ins in the same process, at the same time. We call this approach In-Process Side-by-Side, or In-Proc SxS.

So the .NET Framework 4 allowed Microsoft to ‘innovate in the core’ – in other words, there are new versions of the core libraries. Fair enough, Microsoft do need to be able to make improvements to existing code after all, otherwise we’d still be using .NET 1.0 at the core! But hang on, doesn’t this mean that if you re-build your application in .NET 4.0 that you’ll have to regression test the whole lot because who knows what core type implementations may have changed under the covers? Well no, not really, because of this new approach that Microsoft have adopted called In-Process Side-by-Side execution!

This all sounds great, until you read…

What Does In-Process Side-by-Side Mean to You? > Library Developers and Consumers:

In-Proc SxS does not solve the compatibility problems faced by library developers. Any libraries directly loaded by an application—either via a direct reference or an Assembly.Load*—will continue to load directly into the runtime and AppDomain of the application loading it. This means that if an application is recompiled to run against the .NET Framework 4 runtime and still has dependent assemblies built against .NET 2.0, those dependents will load on the .NET 4 runtime as well. Therefore, we still recommend testing your libraries 
against all version of the framework you wish to support.

Yup, that’s right. If you’re either developing or consuming a library (and really, what developer isn’t doing at least one of those two?) and you decide to upgrade your application to .NET 4, Microsoft actually recommend that you test, test, test!

This tweaked my interest – how different could these new core libraries really be?. I installed dotPeek and opened up both the .NET 2.0 and .NET 4.0 mscorlib.dll assemblies. I drilled down to the Dictionary class and found that there were in fact quite a few, albeit minor, differences between the two implementations!

Anyway, the moral of the story is – upgrading from .NET 2.0 to .NET 3.5 really isn’t a big deal – it’s always wise to do some regression testing, just in case, but you really shouldn’t find many issues with this migration. Upgrading from .NET 2.0 or .NET 3.5 to .NET 4.0 is a different story though. If you’re lucky, In-Proc SxS will help you with backwards compatibility, but if you’re developing or consuming libraries then you’d better have some regression test plans up your sleeve for this migration.

So at the end of the day, it seems like our third-party component bug could well be resolved by rebuilding the whole application in .NET 4 – who would’ve thought!

How to install SQL Server 2008 Management Studio

Posted by Annie on Jun 3, 2011 in Technology | 14 comments

I have a distinct feeling of déjà vu. However, I know I’m not dreaming because I’ve written about installing SQL Server Management Studio before, but last time it was for SQL Server 2005. Why oh why is it so complicated to install this piece of software!?

This time I installed Visual Studio 2010 first, which in turn installed SQL Server 2008 Express. Of course I needed SQL Server Management Studio though, so I downloaded it from Microsoft and tried to install it. And of course, when I went through the installation wizard, the option to install Management Studio didn’t seem to be there. That’s when I start bashing my head against the keyboard and asking ‘why does this have to be so hard!’...

Anyway, after a little investigation and persistence, I figured it all out. I even managed to grab some screenshots so that I would NEVER forget how to do this ever again… and hopefully it will once again help others who may be having the same problems.

1. Install Service Pack 1 for Microsoft SQL Server 2008 – you need Service Pack 1 or else you won’t even be shown the option to install Management Studio.
2. Run the SQL Server Management Studio installer.
3. Click on ‘Installation’ in the left menu.
4. Click on ‘New SQL Server stand-alone installation or add features to an existing installation’.
   
5. It will ask you to install ‘Setup Support files’ -yup, you have to install this in order to install other features, so go ahead and install it.
   
6. Once it has finished installing the ‘Setup Support files’, the wizard should move you onto ‘Installation Type’. As counter-intuitive as it seems, choose the ‘Perform a new installation of SQL Server 2008’ option and click next.
   
7. Now the wizard asks you to specify the edition of SQL Server 2008 to install. You’ll notice that both options given are greyed out – that’s what you want to see. Click next…
   
8. ‘Feature Selection’ – brilliant. This is what you were aiming for. Here you should see a couple of different features that you can install, namely, ‘Management Tools – Basic’. Make sure it is checked then click next.
   
9. Proceed through the next few dialogs, each of which ask you the standard installation questions like installation location, and sooner or later, you should see a nice screen telling you that it has successfully installed ‘Management Tools – Basic’. Success!!
   

HackCamp @ Google London

Posted by Annie on Jun 23, 2010 in Coding, Technology | 1 comment

A couple of weekends ago, I was lucky enough to be able to go along to HackCamp at the Google offices in London together with the mflow team. It’s the first ‘hack’ day/camp I’ve been to – and although I didn’t stay for the whole weekend nor did I actively participate in the hacking, I got to see what a hack camp is all about. For those of you who have never heard of this term before, a hack day/camp is basically a congregation of hackers, developers, designers (geeks in other words) who get together and spend 24~48 hours building an application using whatever tools they have at their disposal, in small groups or alone.

There were about 10 or so different speakers presenting their APIs at HackCamp, and although you didn’t have to use those APIs in particular, it was certainly very interesting hearing about what others have to offer. One of the APIs that I thought was a great idea was this one – 0870.me. 0870.me is an API that allows you to plug in a standard 08* number (which mobile networks charge a fortune for here in the UK) and it will return to you a matching non-08* number if it can find one in it’s database. How neat is that??

There were speakers from Twitter, Google AppEngine and Buzz to name a few big ones, but also lots of smaller companies. Tum and Demis from mflow gave a talk about the mflow API.

Once all the hacked-up apps had been submitted, I had a quick look and found this little gem: tron. It is very cool to see that someone has used the mflow API in their app!

Anyway, having had a small taste of these ‘hack’ camps, I may just have to keep my eyes out for another that I can go to coz they sure do look like they could be lots of geeky-fun :)