Apr 23, 2014

WebDeploy Gets Even More Awesome – Profile Specific Transforms

I love WebDeploy – I have ever since I read Scott Hanselman’s post “Web Deploy Made Awesome: If You’re Using XCopy, You’re Doing It Wrong”. Whenever I’m helping teams that build web applications improve their ALM processes, invariable I end up moving them onto Web Deploy. Not only is it an easier and cleaner way to deploy, but you get the bonus of being able to manage configuration files (Web.config) in your project.

Config as Code

Maturing in ALM means that you need to head towards continuous deployment. You need to be able to deploy your application with a single click of a button. However, one challenge with this is managing configurations. How do you manage Web.config files in dev, UAT and Production environments? Do you find yourself copying config files out the way before you deploy? That’s BAD. You should be managing your configs as if they were code. They need to be source controlled.

Fair enough, I hear you say. So you’ll just copy all your config files from all your servers into a folder and manage them from there. Better – but still lots of pain. A far better approach is to use config transforms.

Config transforms came into web applications in VS 2010. With the latest release of VS 2013, not only are they also available for web sites (NOTE: please don’t use websites – always use web applications!) but you can new preview your transforms from VS and there’s now support for profile-specific transforms.

Let’s have a look at a config transform in a newly created MVC web application. You can see that the Web.config file expands to show 2 other files – one per project configuration:

image

If you add project configurations, then you can right-click the Web.config and select “Add config Transforms” and VS will create a new transform file for you.

Let’s have a look at the Web.Release.config (I’ve removed the comments that are auto-generated):

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<system.web>
<compilation xdt:Transform="RemoveAttributes(debug)" />
</system.web>
</configuration>

You can see that in the <compilation> tag there’s an xdt:Transform to remove the debug attribute. When applied, the transform uses the structure of the transform file to find the correct element and perform the necessary transforms – whether it’s an insert, a remove or some other transform. You can read about the syntax for transforms here. Note that when you debug out of VS, your application will use the Web.config file irrespective of if you’re running in Debug or Release – the transforms only happen when you publish or package your web app.


This is all existing “2012” stuff. What’s new in the latest release?


Preview and Profile Specific Transforms


You can now preview your transform by right-clicking on the transform file (Web.Release.config, for example) and selecting “Preview Transform”. This immediately shows a diff – the original Web.config on the left and the transformed config on the right:


image


Now you can debug your transforms!


So let’s imagine you have a UAT environment. You want to publish your site, so you create a new project configuration called UAT. You then copy the release transforms and put in your UAT connection strings and so on. After doing some testing in UAT, you realize that you actually need to debug, so you’ll have to update your Web.UAT.config file again, or create a UAT-DEBUG project configuration. Soon you’ll get lots of project configurations, and that’s just a mess.


However, you can now create a profile-specific transform on top of the project configuration transforms! Let’s create a publish profile. I’ll right-click the web project and select “Publish”. I’ll create a new profile called “UAT-Debug” and give it whatever settings I need to. On the “Configuration” page, I’ll change the configuration from Release to Debug.


image Then I’ll click on “Close” to save the profile without publishing. That will create a pubxml file for me. I’ll repeat the process and create a publish profile called “UAT-Release”. Looking under the Properties of my project I’ll see the pubxml files:


image





Now I can right-click a pubxml and select “Add Config Transform” – which creates a new transform for me that’s tied to this publish profile.


image I’ll create one for each profile. You’ll see that I now have 4 transforms (but still only 2 project configurations – hooray!)


image


Let’s insert an attribute in both of the new config files:

<appSettings>
<add key="Profile" value="UAT-debug" xdt:Transform="Insert"/>
</appSettings>

I’ve also removed the debug attribute from the compilation tag on the UAT-Release config and removed that transform from the debug one:


image


Now when I right-click Web.UAT-Release.config and select “Preview Transform” I can see the diff. Note that the right-hand file tells me that it has actually applied 2 transforms: first the Release transform (since the profile specifies Release for the project configuration) and secondly the Web.UAT-Release.config transforms themselves.


image Voila! I can now maintain 2 project configurations (Debug and Release) and have a publish profile and transform config per project config and environment. Awesome!


If you’re using Release Management, don’t forget to check out my post about how to use config transforms for parameterizing releases!


Happy transforming!

Apr 17, 2014

Application Insights Telemetry for WAWS or Customer-Hosted Sites Without MMA

There are a couple of tools that change the development landscape significantly for .Net developers. One such tool is IntelliTrace – this revolutionizes how you debug production issues. Another game-changing tool is Application Insights.

There are a few broad categories for Application Insights telemetry – APM (Application Performance Monitoring), Availability, Usage and Diagnostics. In order to leverage APM and Diagnostics, you need to install Microsoft Monitoring Agent (MMA) on your IIS server. This isn’t a problem if you’re using an on-premises IIS or using your own VM in the cloud. However, if you’ve got a website that’s running on Windows Azure Websites (WAWS) or your site is going to be hosted on someone else’s tin or cloud, then you can’t install MMA. However, not all is lost – you can still get Availability and Usage telemetry for these scenarios.

Add Application Insights to Your Web App

Normally this is as easy as installing the Application Insights VS Extensions, right-clicking your Web Application and selecting “Add Application Insights Telemetry”. This adds an ApplicationInsights.config file to your project that allows you to configure Application Insights. This works great in conjunction with MMA. Of course, the scenario we’re talking about is one where there is no MMA.

This scenario is supported, but there’s not a lot of guidance on how to do it. In cases where I’m running my site on WAWS or I’ve created a website that my customer is going to be hosting on their infrastructure (be that on their own premises, their own public cloud or even on Azure) and there’s no MMA, you can still get some really good telemetry. I also had the goal of minimizing admin and configuration footprint as much as possible, since I don’t want to have to maintain lots of config files on other people’s infrastructure. Happily, you can get a lot of good telemetry with a single application setting. Let’s see how to do it.

Create an Application in Application Insights

For the WAWS (or customer-hosted website) scenarios, you need to create an application in Application Insights. This will generate a unique (guid) key for the application. This is the only piece of config you really need.

The first thing you need to do is “Choose Your Adventure” in Application Insights. There isn’t an adventure path for the scenario we’re trying, so we’ll hijack the Web Service adventure. Log into your VSO account and go to Application Insights. Click on “Overview” and then “Add Application”. Follow the prompts along the following path:imageAppInsights will give us a list of instructions at this point. Since we’re going off-track on this adventure, we’ll ignore most of the instructions – we really just need the Application ID.

Type a name for your application into “Step 2” and click “Create”. Then copy the guid from Step 4 (this is the application ID we need to tie AppInsights events to this application in AppInsights):

image 

Note: You can get the application ID at any time by clicking on the gear icon in the top right of AppInsights. Then click the “Keys & Downloads” tab, select your application in the drop-down on the top left and scroll down to the “Web Services SDK” section. The guid there is the one you need for your application:

image 

Follow the instructions in Step 3 – you can follow them here if you like. Select your web application in Solution Explorer, right-click and select “Manage NuGet packages”. Then type in “application insights” (including the quotes) and install the Application Insights SDK package.

image Once that’s done, you can add the following code to your Global.asax.cs file:

var appInsightsKey = ConfigurationManager.AppSettings["AppInsightsID"];
if (string.IsNullOrEmpty(appInsightsKey))
{
ServerAnalytics.Enabled = false;
}
else
{
ServerAnalytics.Start(appInsightsKey);
}

Now add a key to your application settings in your Web.config:

<add key="AppInsightsID" value="guid"/>

The value for the key is the guid you copied earlier when you created the Application in AppInsights. If you deploy this to another customer, create a new Application in AppInsights and update the key in the config file. That’s it.


Setup Usage Telemetry from the Browser


To get usage telemetry from the browser (like Language, OS, Browser Type, Geographic Region etc.) you need to add some javascript to the pages you want to track. Back in Application Insights, you can click on the “Usage” tab. Make sure your new application is selected on the top left. Then click “Web site” for the type of application you want usage stats from. Click on “Click here to show instructions” at the bottom.


image


If your site is on WAWS, you’ll want to create a ping test when you’ve published your application – if it’s already in Azure, then go ahead and enter the site url for the ping-test. This is a synthetic monitor that can give you a bit of basic availability information about your site. Of course when it’s on your customer’s infrastructure there may not even be a publically accessible url.


What’s important for now though is that you’ll see the usage insights snippet in step 3. You can copy that code if you like – though I made a slight modification to this code to make the application ID a parameter, rather than a hard-coded value.


image


I’ve got an MVC application, so I’ve added the usage snippet javascript (mixed with a little Razor) into the _Layout.cshtml page:

<script type="text/javascript">
var appInsightsKey = "@System.Configuration.ConfigurationManager.AppSettings["AppInsightsID"]";
if (appInsightsKey !== null || appInsightsKey.length > 0) {
window.appInsights = { queue: [], applicationInsightsId: null, accountId: null, appUserId: null, configUrl: null, start: function (n) { function u(n, t) { n[t] = function () { var i = arguments; n.queue.push(function () { n[t].apply(n, i) }) } } function f(n) { var t = document.createElement("script"); return t.type = "text/javascript", t.src = n, t.async = !0, t } function r() { i.appendChild(f("//az416426.vo.msecnd.net/scripts/ai.0.js")) } var i, t; this.applicationInsightsId = n; u(this, "logEvent"); u(this, "logPageView"); i = document.getElementsByTagName("script")[0].parentNode; this.configUrl === null ? r() : (t = f(this.configUrl), t.onload = r, t.onerror = r, i.appendChild(t)); this.start = function () { } } };
appInsights.start(appInsightsKey);
appInsights.logPageView();
}
</script>




You can see how I added a C# call (using Razor) to read the applicationID from the web.config – it’s the same call that the code in Global.asax.cs makes.


Custom Events


So that will “buy” me some usage telemetry as well as page hits – all for free (almost). However, I wanted to add some specific events and even do some timing on events so that I can monitor performance. To do that I had to add some code into my methods.


You can log an event (just give it a name – the name allows a hierarchy separated by “/”). You can log a timed event – call StartTimedEvent() with the same naming convention. Later, you call event.End() or event.Cancel() to stop timing. Furthermore, you can log these events with metrics (and/or properties) using a Dictionary<string, object>. Here are some snippets of stuff you can do:


Create a Timed Event for a Controller method:

var aiEvent = ServerAnalytics.CurrentRequest.StartTimedEvent("DummySite/Index");

try
{
// do stuff
aiEvent.End();
}
catch(Exception ex)
{
// log stuff
aiEvent.Cancel();
throw;
}

return View();

Here’s just logging an event:

public ActionResult Contact()
{
ServerAnalytics.CurrentRequest.LogEvent("DummySite/Contact");
ViewBag.Message = "Your contact page.";

return View();
}

Here’s an event with some properties:

public ActionResult About()
{
var even = DateTime.Now.Second % 2 == 0 ? "even" : "odd";
var props = new Dictionary<string, object>() { { "mod", even } };

ServerAnalytics.CurrentRequest.LogEvent("DummySite/About", props);

ViewBag.Message = "Your application description page.";

return View();
}

Of course you can add whatever custom events you need – the beauty of the API is that once you’ve called the Start method (in the Global.asax.cs) you don’t need to worry about IDs or config at all.


Now just publish and start monitoring! (Don’t forget to set your ping-test in the Availability tab in AppInsights if your site is publically available and you haven’t done it before).


Viewing the Data in Application Insights


When you debug your application locally, you can check you events in the “Diagnostics->Streaming Data” page on AppInsights.


image


Once you’ve published your application, you’ll see events start flooding in.


In the Usage->Features->Events page, you’ll see your events, organized into the hierarchy you specified with the / notation:


image When I first saw this page, I couldn’t figure out where the timing for the timed events was. You can click on the little arrow (left of the pin icon for each event) or click the “DETAILS” link on the top left of the graph.


image


Clicking on the “dummysite/index” event arrow takes me to the details for that event, where I can analyze the timings:


image


Clicking on the “dummysite/about” event, I get to see the properties (in this case, “mod” is the property and the values are “even” or “odd” – you can filter the graph by a particular value):


image The Usage->User page shows user/session specific telemetry – mine aren’t all that interesting since I only logged on with 1 user.


image


Clicking on Usage->Environment gives me some telemetry around browsers, OSs, locations and so on.


image


All in all, it’s a LOT of data for only a couple of minor edits to code. You can also add *some* of this data to Dashboards, so you can make custom Dashboards for monitoring your sites.


Best of all, if I give this application to many different customers, all I need to do is supply them with a new AppInsightsID for their web.config file and I’ll instantly get telemetry. Very cool!


Happy monitoring!

Apr 14, 2014

Project Plans, Agile, Pizza and Startups

Last week I posted about how to integrate TFS and Project Server “manually”. In the post I did put in a bit of philosophy about why I think project plans can be a Bad Thing. Prasanna Adavi posted a thoughtful comment on my philosophy, and I wanted to reply just as thoughtfully, so I decided to add the reply as a new post rather than just reply to the comments inline.

Here is Prasanna’s comment:

“I disagree with you on your notion of how "project plans" work. It is wrong to assume that just because a project manager plans and baselines a project, they will never veer from it. To the contrary, It is an attempt to make the customer think before hand as to what they perceive as the value, and then think twice before they "change" the requirements. A good project manager is never afraid to reset the baseline if it is necessary to do so.

Moreover, who said that a "waterfall" has to be one huge waterfall? Depending upon the nature of the project (especially Software Projects), they can be scheduled as multiple mini-waterfalls, yielding quick, unique features. Actually, the way I see Agile is multiple mini-waterfalls. But within each waterfall, there needs to be a commitment. Leaving it open ended and reworking things again and again, because customer does not know what he wants, is not really 'embracing change'.

Think about this: When you order a pizza, do you 'commit' to certain toppings, or do you say lets start making the thing, and we will add toppings as we see the value?

And finally, If your entire team or company is working on one software project/feature set, then it is great to think in terms of "Agile". But when you are on a 'agile' project that never ends because a customer constantly keeps seeing 'value' in the latest change, that is not a project anymore, and it becomes quite difficult to commit to anything else.”

 

Response

Of course when I speak about Project Managers and Project Plans, I generalize. Some Project Managers are obviously better than others. In my experience, however, Project Managers that manage software projects still use the wrong paradigm when they produce a plan – and the fundamental paradigm of a Project Plan is that we know what we’re building and we know how long each task is going to take. If either of these suppositions is incorrect, then the Project Plan’s use diminishes dramatically. Of course in Software Development, it’s nearly always the case that at the beginning of a project (when we’re supposed to be creating the plan) that requirements cannot be known fully – and we can guess how long each task will take, but it’s just that – a guess.

I like to show a diagram called the “Cone of Uncertainty” when I talk about Project Planning:

image 

The Cone shows how the larger the “thing” we’re estimating is (and the further in the future the end date will be), the more uncertainty there is in that estimation. So planning something small right now has little variance – we might say that changing a table in a database will take 2 hours (with about half an hour variance). But if we plan something big (like a Project) and estimate 3 months, then the variance is likely to be +- 1.5 months. We just don’t know enough now to be more accurate.

This is why Agile is about working in small sprints (or iterations) between 1 and 4 weeks in length. That time period is a good fit for the kind of estimations that generally apply to detailed tasks on a software project. Planning further ahead (in detail) is simply wasted effort. Of course, if you don’t have any plan further out than your current sprint, that’s a bad thing too – you need to be working towards some larger goal or initiative. So Prasanna is theoretically correct that Agile is a series of mini-waterfalls – in the sense that the larger Project is divided up into smaller iterations. However, to put a lot of effort into Planning all the iterations in detail is simply a waste of time – things change so rapidly that by the time you’re half-way into iteration 1, the plan for iteration 3 is obsolete.

So being Agile doesn’t imply “not planning”. It just says, to use the 80/20 principle, put 80% of your planning effort into the 20% that matters most – in this case, that’s the current sprint. So plan in detail your next sprint, but wait till you’re nearing the end of sprint 1 before you plan sprint 2 and so on.

Let’s imagine that you’re on a team working in 2 week sprints. Now if you’re only planning in detail 2 weeks at a time, do you really need a Project Plan? Isn’t a list of requirements and their tasks (a backlog) enough?

The point I was making on my previous post was that putting a lot of effort into a large and detailed project plan, and then artificially sticking to that plan, is a waste of effort. And if you’re going to change your baseline often, then why baseline in the first place? Rather plan in detail in the short term (the next sprint) and let changes that come from that spring influence the detailed plan for the next.

Another point is that when you adjust your sprint based on feedback, then you’re “responding to change”. Normally a baseline changes because things are going to take longer than you expected – so the change isn’t based on change (or value), it’s based on poor estimates.

Commitment

Of course just because you’re an Agile team that can respond to change rapidly, that doesn’t mean that you’re at the mercy of the customer’s whims. One of the principles of Agile is to lock down the sprint. So if the customer does decide to change things, they’ll have to wait till at least the next sprint. And of course, since the Product Owner (who represents the customer) is the one that needs to prioritize work, they’ll have to bump something off the backlog if a new change or requirement is added in. In fact, I’d go so far as to say that if your team is doing Agile correctly, you’ll get fewer change requests from your customers and they’ll soon learn to be a lot clearer about what they want and be more sure that they actually want it. Having a detailed Project Plan isn’t necessarily going to make your customers think more about what they want – even if baselines move. But giving them responsibility, having them see that every change shifts the goal-posts, will mean they will feel the change much more “personally”. Also if you deliver something (even if it’s small) at the enf of every sprint, customers get a feeling of momentum. You may even find that the seemingly small piece of functionality that you deliver is enough, and the customer doesn’t really need the rest of The Thing that you would have been working on for the next 2 months. Rapid feedback on value delivered is far better than explaining why a baseline has to move.

Pizza or Degree?

Finally, the Pizza analogy. Again, if you go back to the Cone of Uncertainty, there is very little “risk” in getting toppings wrong on a pizza – so of course you commit to the entire pizza when you order it. I think the analogy doesn’t really work. I think a better analogy is a startup company.

Let’s imagine 2 startups – one called ProjectX and one called AgileX. ProjectX spends 4 weeks getting a detailed plan in place for their product, which is going to take 3 months to build. AgileX in the meantime spends 4 days on a general direction, and a day in detailed planning for their project. They release their product’s V1 after another 2 weeks. The product isn’t fully featured, but what there is of the product works. They get some feedback which changes some of what they know about the product, and adjust accordingly. If they continue to add small features and get feedback in 2 week cycles, they would have released features 15 times before ProjectX’s product comes out the door. They would have had 14 opportunities for feedback from their Customers before ProjectX even had 1 round of feedback on working software. Even if ProjectX followed Prasanna’s “mini-waterfalls”, there’d be less movement. A mini-waterfall would require too much planning for 2 weeks, so ProjectX decides on 3 1-month mini-waterfalls. That still means that they only release 3 times in the same period, and only get feedback on 2 occasions. Still way less than AgileX. Is the more detailed longer-termed planning helping or hindering?

What do you, dear reader, think of Project Planning?

Happy planning!