Apr 14, 2014

Project Plans, Agile, Pizza and Startups

Last week I posted about how to integrate TFS and Project Server “manually”. In the post I did put in a bit of philosophy about why I think project plans can be a Bad Thing. Prasanna Adavi posted a thoughtful comment on my philosophy, and I wanted to reply just as thoughtfully, so I decided to add the reply as a new post rather than just reply to the comments inline.

Here is Prasanna’s comment:

“I disagree with you on your notion of how "project plans" work. It is wrong to assume that just because a project manager plans and baselines a project, they will never veer from it. To the contrary, It is an attempt to make the customer think before hand as to what they perceive as the value, and then think twice before they "change" the requirements. A good project manager is never afraid to reset the baseline if it is necessary to do so.

Moreover, who said that a "waterfall" has to be one huge waterfall? Depending upon the nature of the project (especially Software Projects), they can be scheduled as multiple mini-waterfalls, yielding quick, unique features. Actually, the way I see Agile is multiple mini-waterfalls. But within each waterfall, there needs to be a commitment. Leaving it open ended and reworking things again and again, because customer does not know what he wants, is not really 'embracing change'.

Think about this: When you order a pizza, do you 'commit' to certain toppings, or do you say lets start making the thing, and we will add toppings as we see the value?

And finally, If your entire team or company is working on one software project/feature set, then it is great to think in terms of "Agile". But when you are on a 'agile' project that never ends because a customer constantly keeps seeing 'value' in the latest change, that is not a project anymore, and it becomes quite difficult to commit to anything else.”

 

Response

Of course when I speak about Project Managers and Project Plans, I generalize. Some Project Managers are obviously better than others. In my experience, however, Project Managers that manage software projects still use the wrong paradigm when they produce a plan – and the fundamental paradigm of a Project Plan is that we know what we’re building and we know how long each task is going to take. If either of these suppositions is incorrect, then the Project Plan’s use diminishes dramatically. Of course in Software Development, it’s nearly always the case that at the beginning of a project (when we’re supposed to be creating the plan) that requirements cannot be known fully – and we can guess how long each task will take, but it’s just that – a guess.

I like to show a diagram called the “Cone of Uncertainty” when I talk about Project Planning:

image 

The Cone shows how the larger the “thing” we’re estimating is (and the further in the future the end date will be), the more uncertainty there is in that estimation. So planning something small right now has little variance – we might say that changing a table in a database will take 2 hours (with about half an hour variance). But if we plan something big (like a Project) and estimate 3 months, then the variance is likely to be +- 1.5 months. We just don’t know enough now to be more accurate.

This is why Agile is about working in small sprints (or iterations) between 1 and 4 weeks in length. That time period is a good fit for the kind of estimations that generally apply to detailed tasks on a software project. Planning further ahead (in detail) is simply wasted effort. Of course, if you don’t have any plan further out than your current sprint, that’s a bad thing too – you need to be working towards some larger goal or initiative. So Prasanna is theoretically correct that Agile is a series of mini-waterfalls – in the sense that the larger Project is divided up into smaller iterations. However, to put a lot of effort into Planning all the iterations in detail is simply a waste of time – things change so rapidly that by the time you’re half-way into iteration 1, the plan for iteration 3 is obsolete.

So being Agile doesn’t imply “not planning”. It just says, to use the 80/20 principle, put 80% of your planning effort into the 20% that matters most – in this case, that’s the current sprint. So plan in detail your next sprint, but wait till you’re nearing the end of sprint 1 before you plan sprint 2 and so on.

Let’s imagine that you’re on a team working in 2 week sprints. Now if you’re only planning in detail 2 weeks at a time, do you really need a Project Plan? Isn’t a list of requirements and their tasks (a backlog) enough?

The point I was making on my previous post was that putting a lot of effort into a large and detailed project plan, and then artificially sticking to that plan, is a waste of effort. And if you’re going to change your baseline often, then why baseline in the first place? Rather plan in detail in the short term (the next sprint) and let changes that come from that spring influence the detailed plan for the next.

Another point is that when you adjust your sprint based on feedback, then you’re “responding to change”. Normally a baseline changes because things are going to take longer than you expected – so the change isn’t based on change (or value), it’s based on poor estimates.

Commitment

Of course just because you’re an Agile team that can respond to change rapidly, that doesn’t mean that you’re at the mercy of the customer’s whims. One of the principles of Agile is to lock down the sprint. So if the customer does decide to change things, they’ll have to wait till at least the next sprint. And of course, since the Product Owner (who represents the customer) is the one that needs to prioritize work, they’ll have to bump something off the backlog if a new change or requirement is added in. In fact, I’d go so far as to say that if your team is doing Agile correctly, you’ll get fewer change requests from your customers and they’ll soon learn to be a lot clearer about what they want and be more sure that they actually want it. Having a detailed Project Plan isn’t necessarily going to make your customers think more about what they want – even if baselines move. But giving them responsibility, having them see that every change shifts the goal-posts, will mean they will feel the change much more “personally”. Also if you deliver something (even if it’s small) at the enf of every sprint, customers get a feeling of momentum. You may even find that the seemingly small piece of functionality that you deliver is enough, and the customer doesn’t really need the rest of The Thing that you would have been working on for the next 2 months. Rapid feedback on value delivered is far better than explaining why a baseline has to move.

Pizza or Degree?

Finally, the Pizza analogy. Again, if you go back to the Cone of Uncertainty, there is very little “risk” in getting toppings wrong on a pizza – so of course you commit to the entire pizza when you order it. I think the analogy doesn’t really work. I think a better analogy is a startup company.

Let’s imagine 2 startups – one called ProjectX and one called AgileX. ProjectX spends 4 weeks getting a detailed plan in place for their product, which is going to take 3 months to build. AgileX in the meantime spends 4 days on a general direction, and a day in detailed planning for their project. They release their product’s V1 after another 2 weeks. The product isn’t fully featured, but what there is of the product works. They get some feedback which changes some of what they know about the product, and adjust accordingly. If they continue to add small features and get feedback in 2 week cycles, they would have released features 15 times before ProjectX’s product comes out the door. They would have had 14 opportunities for feedback from their Customers before ProjectX even had 1 round of feedback on working software. Even if ProjectX followed Prasanna’s “mini-waterfalls”, there’d be less movement. A mini-waterfall would require too much planning for 2 weeks, so ProjectX decides on 3 1-month mini-waterfalls. That still means that they only release 3 times in the same period, and only get feedback on 2 occasions. Still way less than AgileX. Is the more detailed longer-termed planning helping or hindering?

What do you, dear reader, think of Project Planning?

Happy planning!

Apr 10, 2014

Integrating TFS and Project Server – Two Way Manual Sync

I often do road-shows showing off TFS and VS to customers around South Africa. Usually I’m doing this with Ahmed Salijee, the Developer Platform Specialist (DPS) for Developer Tools in Microsoft South Africa. Ahmed is an amazing speaker (we’ve co-presented regularly) and is great at helping customers at a strategic level – and, as he likes to say, for his sins, he gets to help customers with their licensing queries!

Ahmed and I agree on most aspects of Application Lifecycle Management (ALM) using TFS – but one of the places we disagree on is the integration of TFS and Project Server.

Philosophy: Why You Shouldn’t Be Using Project Server in the First Place

If you don’t care to wax philosophical about Project Server, then skip this section. However, I think it’s important to step back and think about what you’re getting into if you’re planning on using Project Server for tracking software development projects.

Project Plans are made for environments where “change is bad”. Think about what Project Managers do – they create a plan, perhaps entering in requirements, breaking those down into tasks with estimates that they then farm out to team members. Then they do some curious things: they set milestones and they baseline the project. Let’s examine what milestones and baselining mean.

Milestones are points along the way that Project Managers use to answer a simple question: are we conforming to the plan? Baselines communicate an idea: “What we’ve planned is what we value – we should not deviate from this plan. If we do, then it’ll Be Bad”. Project Plans work well when tasks are exactly predictable and repeatable. Unfortunately, that’s not the case in software development. There are many reasons why this is the case – requirements are language-based, and as such are subject to misinterpretations. Coding is an abstract art – and as such it’s extremely difficult to estimate how long a task will take with any accuracy. Even if you get the code bits right, there are always unforeseen issues in integration of components, deployments, testing and so on. This is why people lie about progress – who wants to deviate from the baseline when the baseline is the Ultimate Good? We’d rather bend the truth about how far we are so that, on paper at least, we look good.

That’s why Agile has come to the form. The basic philosophy of any Agile technique is embrace the change. If we know things are going to change, then why not embrace the change? Let’s shorten cycle times so that we can get more rapid feedback – that way we minimize the risk of doing the wrong thing for too long. Let’s focus on measuring what value we deliver to business, rather than tracking conformance to a plan. Let’s make it concrete: imagine two teams. Team A sticks to the Project Plan closely, and after months of work deliver a system that is average (statistics show that most waterfall-based projects don’t even reach delivery, so we’re being optimistic about Team A’s delivery). Team B deviates widely from their Project Plan, deliver small components frequently and at the end of the project have exceeded expectations. Which team is “better”? The one that stuck to the Plan or the one that Delivered Value? And yet, if we really care about delivering value, why do we beat on teams to Stick To The Plan?

TFS and Project Server Integration: Good and Bad

So you’re insisting on integrating with Project Server anyway. Fair enough. Let’s examine the TFS to Project Server integration.

The good: avoid double entry. That’s about it. The integration allows Work Items added in TFS to be synced over to Project Server (or vice-versa). The integration simply means you don’t have to add (or update progress) in two places.

Perhaps I could add that once you have tasks in a project plan you can do resource leveling and all the other “voodoo” that Project is capable of, but frankly I think that that’s a waste of effort. Let’s imagine you’ve spent a couple of days doing all the leveling. After 3 days, Bob gets sick and is off for 2 days. So you re-level everything, carefully watching as your project drifts from the baseline. Next thing Joe comes and tells you he forgot that one of the changes he’s making will require an extra 2 days of refactoring. You adjust again, sweating a little as your plan deviates further from the baseline. You call up Frank and tell him he’ll have to work the weekend so that you can catch up. Every time something happens, you’re frantically re-adjusting your project plan. At the end, the actual is so far off the baseline, you wonder why you bothered in the first place.

If you train your teams to self-organize, you can track progress of delivered value (which is a much better thing to measure than “conformance to a plan”), rather than dictate who should be doing what when. As changes come in, you embrace the change and adjust course, smiling because change is now a positive thing – not a shame.

If you’re still insisting on integrating TFS and Project Server, there are some caveats that you’ll need to know about before embarking on the integration:

  1. Installing the Integration will modify your process templates. The integration adds a bunch of fields and a Project Server form to your work items.
  2. You need to configure which PS Projects can be linked to which TFS Team Projects. A PS Project can only be linked to 1 TFS Team Project. If you have a lot of PS Projects (or a lot of TFS Team Projects, or lots of both) you’re going to end up with a lot of integration admin.
  3. You need to configure which work items are synced across – at TFS Team Project level. The integration requires you to tell the connector which Work Items it needs to sync. Again, if you have large amounts of PS or TFS Team Projects, you’re looking at a lot of admin.
  4. Updating tasks in TFS does not fill in the Timesheet in Project Server. TFS has no knowledge of when work is done – only that work has been done. That means that if you’re going to want to do billing from Project Server, your team members are going to end up filling in Timesheets in Project Server. Updating Timesheets in Project Server does sync actuals and remaining work for work items though.
  5. The connector is notoriously hard to debug. If the connector has errors, it can be really hard to track them down.
  6. If you have change approvals enabled on Project Server, a project manager can reject changes made to a plan. Imagine a team member updates a work item, which causes the connector to send the change to Project Server. The project manager then rejects the changes. At this point, the sync engine turns off sync for this work item, and the only way to know is to open the work item and take a look. The history has an entry stating the reject reason, and in order to re-sync this work item going forward, you have to re-enable the sync for this one work item.
  7. You cannot assign multiple resources to a Task in the Project Server Project Plan. TFS only allows one resource to be Assigned To a Work Item at any one time – which means that if you’re used to multiple resources on the same Task in Project Server, you’re going to have to split the tasks.

Two Way Manual Sync

So since the integration is so hard (and fragile), perhaps you can consider this alternative: two way manual sync. This page explains in detail the differences between syncing to MS Project versus syncing to Project Server – what I propose here is a “middle ground” that give you best of both worlds without all the pesky configuration required for the integration extension.

Here are the steps to get going:

  • On Project Web Access (PWA), create a new enterprise project.
  • Open the Project and press “Build Team” and add the resources that will form part of this project.
  • Open MS Project and connect to the new Enterprise Project. Check out the plan for editing.
  • On the Team Tab, click “Get Work Items” and select a query for the work items you want to bring into the plan.

 image

  • (Tip: I normally work in the Iteration Backlog and then hit the “Create Query” button to create the iteration backlog query)

image

  • Select the work items from the query and click Add.
  • Now you can work with the Tasks in Project – leveling resources etc. etc. You can also set a baseline if you want to.
  • (Tip: Establish predecessor relationships. Then select all the rows by clicking the row ID – the leftmost column –  of the 1st task and then shift-clicking the ID of the last task. Then right click and select “Auto Schedule”. This creates the initial Gantt for you).

image

  • Once you’re done, hit Publish in the Team tab to save your changes back to TFS. In this screenshot, I added a new Task at the bottom of the Project Plan and hit Publish. This then brought back the TFS Work Item id (48).

image

  • Now you need to publish the changes to Project Server. Hit File and then click the Publish button.

image

  • This can take a few seconds, so make sure you watch the status bar to see that the publish succeeded.

image

  • When you close the project plan, make sure you check it in.

Now imagine that the TFS team is updating their tasks. To pull those updates in, let’s open the Project Plan again:

  • Go to the Team Tab and press “Refresh”. (Notice in this screenshot that the task actuals / remaining have been updated).

image

 

  • One gotcha: If new tasks were added in TFS, you’ll have to schedule them to see the time estimates (see Task 49 in the above screenshot). It’s a good idea to always hit “auto-schedule” on new (or all) tasks to get new tasks into the Gantt correctly.
  • Now you need to publish to Project Server. Again, go to File and click the Publish button.

Finally, consider when new Tasks are added to the Plan in Project Server by another user.

  • Open the Project Plan from Project Server
  • You’ll immediately see the new tasks
  • Add the Work Item Type column and map the new Tasks to work item types. You can also bring in the Area Path and Iteration Path columns.
  • Now go to the Team Tab and hit Publish.

Other Useful Stuff

I wrote a series of posts about integrating TFS and Project server – you can find the fist post here. Also, you can customize the field mappings between TFS and MS Project (not Server) using the guide in this post. Also, if you’re looking for a Timesheet solution and don’t want to use Project Server, then look out for Imaginet’s soon-to-be-released new Timesheet product.

I also can’t recommend highly enough Donald G. Reinertsen’s The Principles of Product Development Flow – it’ll revolutionize the way you think about delivering value to business.

Conclusion

Philosophically I think software development teams should stay away from Project Server (or Project Plans) entirely – focus more effort on measuring delivered value to business than conformance to a plan. However, this change is cultural (and needs to be pervasive) so I know that there are teams that are still going to have to integrate to Project Server. Before you embark on the long and painful process of using the server integration, consider using my two way manual sync to see how it works.

Happy Project Tracking!

Feb 7, 2014

Fix: Release Management WebDeploy Deployment Fails: Access Denied

If you’re using WebDeploy and Release Management (as you should to release Web Applications) you may hit the following error:

Info: Adding sitemanifest (sitemanifest).
Info: Creating application (Default Web Site/MyWebsite)
Error: An error occurred when reading the IIS Configuration File 'MACHINE/REDIRECTION'. The identity performing the operation was 'DOMAIN\tfsservice'.
Error: Filename: \\?\C:\Windows\system32\inetsrv\config\redirection.config
Error: Cannot read configuration file due to insufficient permissions

Seems that the WebDeploy command can’t access some files in c:\Windows\system32\inetsrv. It may be the irmsdeploy.exe MSDeploy wrapper that I’m using for doing WebDeploy in Release Management (see my post about how to do this), since logging into the machine and running the webdeploy.cmd file manually works just fine.

 

The Resolution

You have to add permissions for the release management agent identity to the folder, but this is a folder who’s owner identity is TrustedInstaller – meaning you have to change the owner to yourself first.

  • Right click the insetsrv folder in c:\windows\system32 and select Properties.
  • Click on the Security tab and click the “Advanced” button:

image 

  • Click on the owner tab and then on the Edit button:

image

  • Select yourself (I logged in as TfsSetup which is in the local admin group on this machine), check “Replace owner on subcontainers and objects” checkbox and click “OK”:

image

  • Close all the dialogs and then right-click the inetsrv folder again and click Properties. Now you can allow read access to the Release Management agent identity to this folder.

Once you’ve changed the permissions, you will need to reboot the machine. After the reboot, the WebDeploy through Release Management should work without a hitch.

Happy releasing!