Please find below a transcription of the audio portion of Fletcher Hearn’s session, Project Performance Measurement – Part 1: Overview Of Project Performance Measurements, being provided by MPUG for the convenience of our members. You may wish to use this transcript for the purposes of self-paced learning, searching for specific information, and/or performing a quick review of webinar content. There may be exclusions, such as those steps included in product demonstrations. You may watch the live recording of this webinar at your convenience.
Kyle: Hello, and welcome to part one of MPUGs Project Performance Measurement course. Part one covers an overview of project performance measurements. My name is Kyle, and I’ll be the moderator today.
Kyle: Today’s session is eligible for one PMI, PDU in the technical category, and the code for claiming that with PMI is on the screen now. Like all MPUG webinars, recording of this session will be posted to mpug.com shortly after the live presentation ends. All MPUG members can watch the recordings at any time and still be eligible to earn the PDU credit. All the sessions you watch on demand can be submitted to your webinar history and the live sessions you attend are automatically submitted within your history. You can print or download your transcript and certificates of completion, including one for today. You can access that by logging into mpug.com, click the My Account button and then click on the transcript link.
Kyle: If you have any questions during today’s presentation, please send those over at any time using the chat question box on the go to webinar control panel. We do plan to answer your questions throughout the session today.
Kyle: All right, we’ll go ahead and get started. We’re very happy to welcome back Fletcher Hearns today. Fletcher is the director of training solutions with Edwards Performance Solutions. He is responsible for the implementation of enterprise project management solutions for Edwards’ clients as well as overseeing the custom application development performed at Edwards around enterprise solutions and Microsoft Project for both desktop and server as well as SharePoint. Fletcher is also one of the leading trainers at Edwards in various aspects of project management and the use of project management tools. With that said, I’d like to welcome you back, Fletcher. I’ll go ahead and pass it over to you to get us started with today’s session.
Fletcher Hearns: Great, thank you Kyle. Thank you so much. Technology is wonderful. Hopefully everybody can now see my screen.
Kyle: Yes, that looks good.
Fletcher Hearns: Perfect. I want to thank everybody for joining me today. It also not only gave me the opportunity to talk about project management, but it actually gave me an excuse for the first time probably in 14 months to actually get dressed up and come to work. I came to my office today because I was having internet issues at home, and so I will ask for your indulgence because our office is actually quite close to a hospital. If an ambulance starts to drive by behind me, I may mute myself for just a couple of seconds to let the ambulance go by so that it doesn’t interrupt our conversation today.
Fletcher Hearns: Again, Kyle, thank you for the wonderful introduction. Today, we’re going to talk about project performance measurement. This is part one of a three-part series we’re doing. Today, I just want to give you an overview of what that’s about and then in the second part we’ll talk more specifics, and in the third part I’ll actually demonstrate how to use a tool like Microsoft Project to generate the metrics, the data analysis you need either within the tool or doing what we all like to do sometimes is export it out to another tool like Excel, and then create the information you need to communicate with your sponsors, your stakeholders, your clients, your customers about how your project is doing.
Fletcher Hearns: With that, I am going to turn off my camera simply so that we can save the bandwidth and hopefully everybody will have a better presentation on this. Oops, [inaudible 00:03:23].
Fletcher Hearns: Again, Project Measurement Performance. This is part one of the series. To give you a little background on myself, my name is Fletcher Hearns. I am the director of technology solutions and CTO for Edwards Performance Solutions. We are a consulting company in the DC Baltimore area that specializes in project and portfolio management for about 80% of our customers in the government space, the other 20 in the commercial space. By education and training. I’m not an educator or a trainer. I actually got my degree in software engineering and moved up into project management like a lot of us did back in the day. Well, you didn’t screw up your last job, so you got promoted to the next job. Individual contributor team, lead section, lead project manager, program manager, director of engineering. Spending about half my time in the commercial space and half my time in the government space, so I kind of got a well-rounded thing on how it’s done in both places.
Fletcher Hearns: As Kyle said, if you have questions, drop them into the chat as we’re going along. I will try to remember to stop and answer questions. But if not, always put them in there. Kyle, please interrupt me if I don’t see people’s questions as I’m going through this, but we will make the best we can to make sure we get all of your questions answered either before or after this presentation.
Fletcher Hearns: One thing I want to talk about and look at is looking at what you should measure, what you can measure and what do you want to measure because sometimes those are indirect conflicts. We’ll talk about what you should measure based on the project you’re running, or what you can measure because you have the data to measure and what you want to measure looking at what is most effective to measure and communicate. Throughout this, I will try and touch on traditional, agile and hybrid as a way of saying they’re all the same, but they’re all very different.
Fletcher Hearns: Hybrid, for those of you that may be new to this, hybrid is, especially in the government state, local federal government units, the way they’re starting to adapt agile, which is they will do part of the project in waterfall and part of it in agile. Normally, they’ll do requirements gathering, system design. High-level software design will all be done in the traditional waterfall then they will spin into an agile environment to do the software development part of it, then they’ll spin back out of Agile to go back and do something like user acceptance testing. System integration testing will be done in traditional waterfall method as well as the rollout. So, that’s what a hybrid is. It is taking both traditional waterfall and agile and pushing them together into a project or a program. That’s what we call hybrid in most cases.
Fletcher Hearns: Why should we measure any of the stuff? Well, looking at the PMI pulse of the profession 2016, 45% of all projects experienced scope expansion or scope creep or gold plating. When they talk about that, that is uncontrolled, meaning it wasn’t a change or a request come in for scope expansion. It was more work was done than was originally planned and it wasn’t a formal change request. Thirty-two percent failed the project’s budget and was lost because of that, and then, overall, 16% of all projects started in the survey failed or deemed failures. That is a huge amount.
Fletcher Hearns: Another quote from that same study was they see a S122 million [inaudible 00:07:10] if you don’t measure project performances, the right information to the right people at the right time will allow effective decision making. So, by measuring something and quantifying things, we can then make decisions based on that, and it’s all part of project management communication.
Fletcher Hearns: There was a quote by another consultant in the DC area that I met with and worked with, and he says, “Managing a project without metrics is like sailing a ship without a compass.” You’re just floundering around. You don’t have any destination, you don’t know where you’re going, and you don’t know how to get there. That’s why we want to measure, because how can we possibly know how well a project is performing if we don’t have certain measurements that tell us how it’s performing so that people can make effective decisions?
Fletcher Hearns: I’ve worked in commercial areas, I said, for about 20 years of the 40 years I’ve been doing this, and in the commercial space it’s not uncommon. They want the measurements because they want to know effectively when should we cancel this product, when should this product no longer be built because there’s no way we’re going to make money on it. So, scrap it and move on to something else. So, that’s done a lot more in the commercial space than it ever is in most government spaces. They tend to not want to cancel projects. They will retool them, they will rescope them but they’re not necessarily into basically killing it.
Fletcher Hearns: A couple things to keep in mind when we’re going through this. If you don’t measure it, you can’t manage it; if you don’t measure it, you can’t improve on it; and if you don’t measure it, it probably means you don’t care about it. The last one is okay. It’s okay to say I’m not measuring that because I don’t care about that measurement for this project, or this program or this portfolio. And the last one is, if you can’t influence it, then don’t measure it. Just because you can get the data doesn’t mean you should do something with the data.
Fletcher Hearns: I worked with a project manager before that, their systems, in the order they were set up, he could get not only the costing for his project and how it was going through the cost accounting systems, he could get them for any project in his organization. He wanted to go through and start looking at how other projects were working, and I’m like, why would you ever look at the cost of someone else’s project and how they’re performing? You can’t influence that project. That’s the project manager, that’s his direct management team that can control that. You measuring that and putting metrics together and say, “Hey, I’m performing better than Joe is,” why? You can’t influence Joe’s measurement, so don’t measure them, don’t report on them.
Fletcher Hearns: What to measure starts at project inception. It starts all the way from when you’re putting a project charter together through communication management, through stakeholder management. It’s all part of that. Looking at what your stakeholders want to communicate. What do your stakeholders care about? What do they want reported? What are their concerns? What does the team care about? What does your team, the people working for you, what do they want to look at and measure to see how well they’re performing? What are the organizational corporate needs?
Fletcher Hearns: Regardless of what you want to measure, you may have organization or corporate needs that say you must measure this on all projects. Whether that’s rate of return, whether that is quality measurements, whether it’s cost, whatever it is, those are right there. Whatever external stakeholders require you to measure for compliance.
Fletcher Hearns: As I said, I spent a lot of time in the government space, and we have to do earned value management not because necessarily the organization wants to, or the team wants to, or any of the internal stakeholders want to, it’s because the government says for this larger project, you will measure and report on earned valued metrics every month or every quarter.
Fletcher Hearns: The last one is what do you need to manage the project? What are the metrics that you want to collect? What are the measurements you want to take so you can be assured that you understand how well the project is progressing? What the performance numbers are, what they mean and, if they don’t fall in line with what you want them, how are you going to change them? Those are the ones you need and those are the ones that regardless of what the four other ones above it are, what are the ones you need to measure to make yourself comfortable as a project manager that you can adequately manage the project?
Fletcher Hearns: And then, establishing the KPIs and the CSF. I think at the next page are probably, yeah, critical success factors or CSF, an element that is necessary for a project to achieve its mission. Those are much more soft targets, if you will. They’re not always directly easily measured. High quality product is implemented and utilized, business objectives are achieved. It’s not that they can’t be measured, they may not be able to be measured as part of the project especially like business objectives are achieved. If you’re going to have a, “Oh, at the end of this project, over the next three years, we expect this cost measurement to go down,” that’s great. But as the project manager, it’s not something you can measure necessarily while the project is going on, and you probably won’t be the person measuring the business objective as it’s achieved because it’s something that’s measured after the project close out.
Fletcher Hearns: But if you look on those, the left-hand side shows you the critical success factors. And then we start to talk, on the right-hand side, of the key performance indicators that you would actually measure to. The first one is the value of the features delivered at completion, whether it’s a project completion or iteration completion in there, is there value of that feature? Is the quality of the feature delivered at completion, project or iteration, does it help us build a high-quality product and it’s implemented in utilize?
Fletcher Hearns: Business objectives are achieved. While you might be able to do things like increase online sales, you can measure the increase sales, increment value by delivery. Is each increment that we might deliver of this project, especially in something like an agile environment, is it adding value to the total product? Is it helping [inaudible 00:13:18] to our business objectives?
Fletcher Hearns: Does the project meet schedule and budget? That’s the critical success factor, it came in on time and on budget. And then the KPIs is cost performance indicators and variances, scheduled performance indicators and variance, whatever other measurements you might use to do that resource utilization, those type of measurements.
Fletcher Hearns: Resource are effectively utilized. Resources are needed versus the plan versus the availability versus the used. Here’s what you said you need, here’s what I planned, here’s how I planned it, here was the availability of resources when I actually needed them, and here are the resources that I actually used on the project. And then team cohesiveness especially in agile environment. Is the team working well as a team? Is team cohesiveness there? Is it something we need to work on to change for the next project or within a project? Especially when you’re talking agile where it is the team that does the work, not individuals as might be called out in a traditional waterfall work environment where it’s not a team working together with milestones and measurements. It is each individuals working on tasks that then get us to milestones.
Fletcher Hearns: And then, project adherence to best practices. Adhere to organization and industry standards. Are they doing it along the way? Is it something we need to review? What are the lessons learned? Are we doing good change and risk management? And then, accurate and relevant project status and forecasting. In other words, are we doing all the stuff that is part of measurement? Measurement is great, but if you don’t do anything with the measurements, what are you doing them for? Creating nice reports that you then stick in the drawer of your desk don’t help anybody.
Fletcher Hearns: So then, are there any questions as we’re going along here? I want to make sure I have time for questions. I’m not seeing any of that.
Kyle: Oh, we did have one question that just came in from Teresa. “[inaudible 00:15:13] this will be addressed later, but I’m just curious about measuring team cohesiveness and the best approach for that.”
Fletcher Hearns: What I’ve done in the past, it is doing surveys. The best way to measure team cohesiveness is to use some form of survey, a survey monkey, a Teams form, something. They just ask questions about that person’s feelings about how it is to work with this team. You can also use interviews, but interviews are a little different because they’re not as anonymous when you’re talking about team cohesion. And people having to talk about people, if they can do it anonymously, it’s the best way to do it. I always say, “No, we’re going to use a survey monkey or a Teams forms or something that says this is completely anonymous.” I don’t know who gives the answers, but we’re getting answers back about how we’re working as a team. That’s the best way. Agile books tell you, no, that all should be done in person and all that stuff, but that really doesn’t work as well because people don’t want to say things about other people sitting across the table from them, that they really might want to get out because of the dynamics of humans. That’s just it. Great question, though.
Fletcher Hearns: What can be measured? Well, each project is different. Metrics can be set with the project type, construction versus software development. Each project has unique requirements. Everybody understands that. But then again, every project is also the same.
Fletcher Hearns: All projects have scope, cost and time built into them. All projects have risks. Sometimes you’re going to use measurements from both, things that are unique to your project. If you’re doing software development in agile, you might be measuring things like product backlog growth. In construction, you might be looking at how many panels of a bridge have been completed over time? Because those are different things you measure for different projects, but both of those projects will have a defined cost, a defined scope, and a defined time. Yes, I know scope in agile gets a little wonky, but there is still some scope especially as you’re doing more and more of it in the backlog and management of it.
Fletcher Hearns: All projects have risks. There isn’t a project on the planet that doesn’t have risks. So, how do you measure risks? How do you look at what the risks are? We’ll talk about that some as we go further into the next episode about how to look at things and how to create ways of doing that.
Fletcher Hearns: And then you also have to understand at what level you’re going to be reporting. Are you reporting at the corporate PMO level? Are you reporting for all projects across your organization? Are you looking at a portfolio of projects? No, I’m only worried about this portfolio projects for IT systems, or I’m worried about only this portfolio project that has to do with us expanding our market share out of the US into the South Asian market, or into the European market. Those could all be different portfolios of projects, and you might be doing measurements across all of those as well as at the project level or at the program level. So, there has to be some understanding of what you’re measuring and at what level those measurements are going to take place because that changes how you measure things.
Fletcher Hearns: You’ll notice here, I’m a big Dilbert fan because Dilbert is great.
Fletcher Hearns: Waterfall, agile and hybrid. People say you can’t measure agile projects the same way as you can waterfall projects. Yes, you can, at some level. There is enough cross compatibility between all projects because, as I say up there, all projects are the same. They all have scope, cost and time, so we can measure those three things across all projects.
Fletcher Hearns: I work with a lot of organizations that are doing agile and certain parts of the organization are doing system development and generating very complex, very customized pieces of hardware for their organization. That is done in a very traditional manner because once you lay out a board, you can’t change it a week later because something changed. You have to lay it out, you have to understand what it has to do, you have to have all those done.
Fletcher Hearns: And how do we measure the success of those projects across? Well, there are certain commonalities that you can use. In the Agile world, you still measure things like velocity. You can still measure other things.
Fletcher Hearns: As I said, while each project is different, there has to be a commonality because at certain levels within organization, I will tell you, when you walk into a CIO, a CEO, a CFO, they want to know several things about the project. They don’t really care normally what the velocity is of your agile software development teams, they don’t care about escaped defects through iterations. What they want to know is I’m on time; is it on budget; and am I getting value for the dollars I’m spending; am I getting what I asked for, for the dollars I’m going to spend? That’s at their levels. So, it gets to be, something has to roll up to allow you at the highest level in an organization to report it; while, at the lowest level, being able to give even more details to make the improvements you need to keep the higher level metrics in place.
Fletcher Hearns: What can be measured? Many things can be measured. Velocity, ROI, critical path, requirements, ETC, BEI, SPI, cost variance, lead times, cycle times. This is about 29 or 30 different metrics that you can measure. You have to determine through all of these which ones can you measure, which ones must you measure, and which one should you measure. I’m sure if you look through here, there are some that you don’t know what they are and there are some that you say are missing. There’s no way to be able to put on one slide all of the different metrics and measurements you could take for an individual project.
Fletcher Hearns: People will say, there’s percent complete and percent duration complete. What’s the difference? One measures basically, percent complete, based on the work that was done and one measures the percent complete based on the task to calendar time. Well, which one should I use? Well, what are you trying to measure? Then I can tell you which one to measure.
Fletcher Hearns: So, we have many things to measure. Now we have to figure out, just because it can be measured, should it be measured? What are you going to measure on your project? You’re going to measure everything you must measure. Everything the stakeholder requires you to measure, everything the organizational requires you to measure, everything you’re contractually required to measure. As I said, in the government space, if you have a contract over a certain dollar value, you as a program project manager may have no choice. You will be required to do earned value management and report on it every month or every quarter. You have no choice. It’s not that you get to do it or don’t get to do it, you are required to do it.
Fletcher Hearns: The stuff you want to measure performance. What are the ones that you as a project manager want to measure above and beyond what may be required? And you must manage it. What are the ones you want to look at? Hey, these are the two or three I want to measure because if I measure those, I know how to use those to make improvements and get the project done on time, on budget with the quality that’s required in the timeframe required.
Fletcher Hearns: Just as importantly, don’t measure everything just because you can. Just because you have the ability to measure all 29 objects I have on board here doesn’t mean you should. If no one is telling you, you have no musts that you must do, pick no more than three to five good measurements. Pick three or four times that you’re really going to concentrate on and measure and then figure out if those numbers, those metrics, those measurements fall outside of the boundaries. Figure out how, if they fall out, I’m going to know how to start working on bringing those back into alignment.
Fletcher Hearns: Make sure, when you go try and create metrics based on measurement, the use of objective and not subjective measurements. When somebody tells you a task is 50% complete, it’s not valid to say it must be because 50% of the dollars I was going to spend on that task had been expanded. Or, well, it’s a two-week task and I’m a week into it therefore I must be 50% done because it’s a two-week task and half of the time is done. Simple. It’s 50% done. Or, the worst case is because some software developer said so. You walked to the software developer, you lean over [inaudible 00:24:29] and said, “Hey, how far done are you on that task?” “Yeah, about 50%.” And then you ask him a week later, 75% late. Great. You ask him a week later, 80%. A week later, 92%. A week later, 93%.
Fletcher Hearns: Those are objective measurements. They aren’t something that you can quantify and say, I know I’m 50% done because the first draft of the document is done. Half of the books that we are going to deliver have been created. Twelve of the 24 architectural drawings we need for the new bridge are completed and signed off on. I have lost 20 of the 40 pounds that I need to lose because of the COVID. A feature has been delivered but not tested and accepted, it’s only been developed. Those second set of things are things that are measurable. I want to say tactile, but they’re not necessarily tactile. But there are something that can be said, I know I’m done because of this.
Fletcher Hearns: One of the most important things for successful measurement of project performance is defining what the meaning of done is upfront for all tasks. This task is done because, and it’s not somebody said so, it’s because I delivered something. I did something. I can say I did something. I know I did something.
Fletcher Hearns: Some way of saying it’s done because the design spec has been approved. All 24 architectural drawings have been approved. I delivered these six features at the end of the iteration and they were all accepted by the product owner. We had acceptance criteria for each of the features that we met all of the acceptance criteria for the features. I hit the development milestone, start development milestone because all of the design specs have been done. I hit the minimum viable product in terms of being able to release because all of these features were completed and each of them was accepted. It has to be something that’s objectively measurable. That is key to a good measurement performance of a project, is having metrics that can be used that are as objective as possible.
Fletcher Hearns: And then, what can be measured. You need to look and understand leading versus lagging indicators. A leading indicator is predictive in nature. The measurement of current events highly correlates to future results, and then drive future results through cause-and-effect relationships. So, leading indicator is saying if this is this, then I can, with some level of confidence, predict what the future performance is going to look like if I make no changes.
Fletcher Hearns: Lagging indicators are results from past events, and the results are based on measurements after the facts and are output oriented. Now, a lot of times, it is the lagging indicators that you initially get that help you figure out the leading indicators you should be measuring. So, a lot of times we’ll start looking at lagging indicators, and if those can’t be used predictively for the future, we then look for the leading indicators to say, well, if that’s what I see now, what do I measure earlier in order to find out what that number is going to be in the end.
Fletcher Hearns: I have a couple examples here I’ll walk through. A couple are non-project related and some are project related. So, if we’re looking at weight loss, the lagging indicator is the scale display. It’s getting on the scale every morning and seeing what the number is. That’s called a lagging indicator. Why? Because it’s after the fact. The leading indicators would be looking at the calorie intake, how many calories do I take in per day, and the calories expanded per day. The simple math says if I have a calorie intake less than or equal to my calorie expanded, the scale is not going to be my friend in the morning. So, the leading indicators are calorie intake and the calorie expended and then the lagging indicator is looking at the scale because once you look at the scale, you can’t make a change for something before that. So, if I want to lose weight, the best way to do it is to do one or two of the leading indicators. Don’t eat as much and exercise more will show up on the scale days or a day or two later. So, that’s how we look at that.
Fletcher Hearns: Another one would be, I’m looking at the truck efficiency if you’re running a fleet of trucks. The lagging indicators are looking at your fuel expenses that are continuing to climb and your miles per gallon are continuing to climb. Those are lagging indicators. That says, hey. Well, what’s the leading indicator say? Hey, if I change this, then those numbers will probably go in the direction I want, and that will be looking at something like your vehicle maintenance. Are you doing oil change every 5000 miles? Are you inflating the tires correctly? Those will help with the lagging indicators. But sometimes an organization picks up on other lagging indicators. “Hey, why are my fuel expenses going up every month and why, based on that and the number of miles that the trucks are traveling, the miles per gallon is going down?” It’s looking at those type of indicators.
Fletcher Hearns: Here’s one for project execution. If you look at the number of change requests, and I’m talking about the number that come in. Not the number that are necessarily approved, but the number that come in. It is a leading indicator that you’re going to have crossover runs in these milestones. What has been found out is if change request come in and people see the change request, if they’re approved or not even approved, but there’s a list of them especially in the software world which I work in. It happens at gold plating, and scope creep occur because the team saw that come in there. They told the guy, “Yeah, if you want that feature, that function, you want that change from blue to red, you got to put in a formal change request. Put the form in there. It’s got to go through the CCC board at the end of the month.”
Fletcher Hearns: What happens is, the developer, if they’re not very disciplined, ends up putting those types of things in. And then the control board says, “No, we’re not changing that from red to blue. It was red in the spec, it’ll be red on delivery and if you want to change, we’ll change it later. But to make our cost deadlines and our milestones, we have to deny this one.” The problem is, it’s already in the software and now you have cost overruns and missed milestones.
Fletcher Hearns: Another one, a leading indicator is to look at tasks starting late. If you see lots of tasks starting late, there’s a good chance you’re going to have missed milestones, late delivery, and resource over allocation issues because when you had the perfect schedule, your resources were allocated to do things at certain time. And if you start that task starting late, you’re now going to have a stack up of tasks and you’ll have resource over allocation. But, when you see the lagging indicators, you have to basically do project management. What’s the root cause? Well, the root cause was late starting tasks. Okay, let’s measure how many tasks are starting late over a given period of time.
Fletcher Hearns: So, those are just some ways of seeing the difference, and you want to get to figuring out, measuring as many leading indicators as you can because if you do that, you’re able to catch and correct the problems before they become missed milestones over allocated resources and late deliveries. You’re able to go back and figure out what those are earlier.
Fletcher Hearns: As I said, though, sometimes if you’re not paying attention, you have to use a lagging indicator and then do the recall and say, oh, the lagging indicator says that I’m having late deliveries. Well, why am I having late deliveries when my resources are over allocated? Well, why are my resources over allocated? Well, because 20 of the 30 tasks that were supposed to start this month all started at least four days late, and so, all the people that were supposed to be working on it now are stacked up [inaudible 00:32:31] on three or four tasks at a time instead of one task at a time.
Fletcher Hearns: Why do we have to measure? Why should we measure project performance? Because we have to. It’s a requirement of the contract you’re on. Because we want to. Because good project managers want to measure performance. Why? So that we can communicate with our stakeholders, our customers, our clients. The good, the bad and the ugly so decisions can then be made. Remember, that’s part of the whole part of measuring project performance, and using all this information is to communicate the good, the bad, the ugly to all those people so that they can make a decision.
Fletcher Hearns: As I said, I worked in commercial businesses, I worked early in my career at a company called Digital Equipment Corporation, which at the time was the second largest computer manufacturer in the world, and they always looked at these measurements because there were times where a product was in development and things like the ROI would not work out. They would just say, “No. That disc drive is no longer to be made. That project is canceled. Everybody would be dispersed onto newer projects,” and it went on. But they use that to make a decision of we can’t make money selling this disk drive because it’s going to take us too long to get to market and there isn’t enough time to sell it before the next generation disc drive comes out.
Fletcher Hearns: And it happens today in the cellphone industry. If you look at the history of the iPhone, or Samsung or any of the other major manufacturers, you will see where they were coming out with phones on a regular basis. Every 18 months they announce a new phone, every 24 months they deliver it. And there’ll be a gap in there because at some point they figured out there was no way to make money on this iPhone version, so they scrap it, pull the next one in six months and then they would have a 30-month or a 32-month gap instead of the 24 because there was a phone in there that made no business sense for them to keep going, and it’s because they were measuring the cost of the product and determined they weren’t going to make any money on it
Fletcher Hearns: So, we can correct. One of things, you measure so that if you have deviations from what you’re expecting, you can make corrections. Earned value management was specifically designed for this. It said, if we start looking at these numbers, we should be able to then predict how the project is going to perform in the future. And if we can guesstimate how it’s going to perform in the future, if it’s not performing well, we can make corrections early on.
Fletcher Hearns: Earned value management, within the first six months, you can start to get really good data on how the future is going to look for that project. You’ll start really measuring it month three, and then by month six you’ll have a good idea. And earned value was always designed for large programs. It wasn’t designed for two-month $20,000 projects. It was designed for sending man to the moon, building aircraft carriers. It was those types of projects they were trying to keep the control on.
Fletcher Hearns: So we can improve. If we’re not going to improve, why are we measuring it? And then, of course, so we can make pretty dashboards, and everything can go ooh and ahh at our quarterly program review meeting. They go, “Ooh, that looks pretty.”
Fletcher Hearns: I want to talk a little bit, just to be upfront, that agile and traditional are very different. I run traditional product research and development, which are very traditional waterfall or working in the government space where it was all done in traditional method, and I’ve run software development efforts that have been all purely in agile.
Fletcher Hearns: In agile, there are different things that you measure. You measure things like velocity, sprint burndown, lead time, cycle times, escaped defects. In a traditional one you will get percent complete, percent duration complete, baseline execution index, resource utilization, late/early task completion. Again, all of them, at least the first three, are metrics you can really measure tightly; cost variance, schedule variance and estimate to complete. Those are the same across all types of projects. So, we can measure those.
Fletcher Hearns: So, while at a project level, at an agile project, we’re measuring the cost variance and the schedule variance. We may also be measuring velocity, sprint burndown, lead times and cycle times, but as it moves up in the reporting, a lot of those are going to go away. I said earlier that the farther you get up the chain of command, the less likely they are to care about what the velocity of a scrum team is. You got three scrum teams working on this project. The CIO might care, maybe won’t care. The CFO is not going to care what the velocity of scrum team A is or scrum team B or C. He is going to care what’s the cost variance running on this project, what’s the schedule variance running on this project, and how much more money ETC do you need to complete the project? That’s all a CFO is going to care about.
Fletcher Hearns: In a traditional one, we have percent complete, which is the percent of the work that’s complete; percent duration complete, the amount of time that’s complete. Baseline execution index is one of the things that you used to track things like late starting tasks, late finishing tasks to figure out how your project’s doing if you’re not doing other types of project where you have fully resource loaded schedules. We’ll talk about that a little more in the next session in a week. They tend to look more on resource utilization on traditional than we do in an agile environment because we put an agile scrum team together. They’re fully committed to that project until that project is done versus on a traditional one where we’re bringing resources in and out as we need them. On a scrum agile software development, you put a team together and you say run with it. They have everybody they need, and they’re dedicated to that project.
Fletcher Hearns: Metrics data versus trend. Metrics data is a point in time measurement. It does not tell a story, it’s just the facts. This Friday, here are the facts. Every Friday I have a set of facts. They don’t really tell a story. They just say here’s what it looks like on this Friday.
Fletcher Hearns: Trend analysis says, take those metrics and look at their progression over time to tell the story. So, it’s when you have to look at not only the individual ones, but the trend of how [inaudible 00:39:12], and that’s done in almost all projects today. For instance, metrics are collected for a week. The velocity for this at the end of this iteration is 22, the CPI for this project is 0.75, the SPI is 1.03, the cycle time is 3.2, and defect for iteration is 2. Those are all great metrics and those are all things you probably should collect. But if I just give you that, you cannot make a decision on how the project’s doing. What you can tell me is, yes, for project A, CPI is 0.75. But you can’t say how it’s doing because you have nothing to compare it against. This doesn’t tell us anything about how the object is doing. This doesn’t tell us how we will do going forward, and that’s when you want to look at trending.
Fletcher Hearns: I know I haven’t stopped in a while. Kyle, if there are questions, please interrupt me.
Kyle: Nothing in the queue just yet. But just as a reminder to anyone who does have questions, feel free to chat those over and we’ll take those live throughout the session or at the end of the session today.
Fletcher Hearns: Here, I pulled together two basically trending charts that can be used for measurement reporting. They show the metrics and measurement over time and they allow you to look at what the trend is to see what the future performance might look like.
Fletcher Hearns: On the left-hand side, I pulled one out for the velocity. Now, I don’t know how many people here are familiar with agile. If you’re not, I’m going to give you the two-minute overview of what agile and what velocity is.
Fletcher Hearns: Agile is a project management methodology that basically breaks your project down into very short iterations, usually in the order of two- to four-week iterations. Basically, for lack of better term, if you’re not familiar with agile but you are familiar with what, in traditional project management we call it a rolling wave way of doing project management where you do six-month chunks of real detailed scheduling, costing, tasking. Well, think of doing that in two-week intervals. Think of it as rolling wave on [inaudible 00:41:27]. Instead of doing six months of planning, we’re doing everything for two weeks. And one of the measurements you use in agile if you’re doing software development is what’s called the velocity which, and for you agile people on this meeting, I apologize, I’m going to butcher agile, just make it as simple as possible, it is the amount of work you’re getting done per iteration and it’s measured in what’s called story point.
Fletcher Hearns: So, this team here on the left-hand side, each week, the velocity, which on the first week looks like it was about 12 and in the third week it looks like it was up as high as 27 or 28. That is the measurement of story points that they completed. Think of a story point as a piece of work for us pure people in the thing, amount of work they got done in a given iteration. So, in the first iteration, they got 12. The second one, it looks like they got 17. The third one, they got 27. The fourth iteration, they got 14, and so on.
Fletcher Hearns: But what you can see over time is they have settled into somewhere right around 25 story points per iteration. So, the trend is about 25. I can use that as a measurement, and I’m tracking it. And you’ll see deviations from about iteration 9 through 17. It’s been pretty steady right around 25. It’s like, in the very beginning, they’re trying to figure out what they can do. It’s a whole bunch of things. But the first three to four weeks, you really can’t use those as a good trend. Then you can start to see how it’s trending and you see it getting in there. If I start to see this over time, in the next five or six weeks, start to dip below 25, it’s an indicator that there’s something changed with that scrum team in getting their work done. It’s something I’m going to start looking to investigate.
Fletcher Hearns: The right-hand side chart is a traditional earned value chart. Basically, it’s saying we are 12 weeks 12 months into an 18-month project. Our planned value, how much we should have spent, how much value we should have gotten out of this would have been $3.8 million worth of value. We actually spent 3.2 million in actual costs and we got 2.7 million of value. I’m not going to go into a whole earned value management. That’s a whole different thing I could go through. But with this trending, as you can see, if the blue line is what we should have done, the red line measures what we actually did in terms of cost, and the green line measures the value of the work we got done.
Fletcher Hearns: You can see, we’ve never been well. In a perfect project, the blue line, the green line, and the red line would all overlap. And so, we can tell here that the trend is not good. It’s actually flattening out. If we are to try and make this number up here at the very right-hand side where, in 18 months, we should have sent for $4.9 million to get this project done and the green line represents how quickly we’re getting work done, there is no way from the end green point to the end blue point we’re going to get done if the trend line continues, if we make no changes to how we’re managing the project.
Fletcher Hearns: So, this is when the past performance and the past measurements can help guide you to say if we don’t make a change, here’s where we’re going to be at the end. The same with the velocity. If they can do 25 story points per iteration, and I know there are 200 story points left to be done or 200 points worth of work to be done, I know that’s eight iterations and this project is done if nothing changes. Where, if it starts to go down to 20, I know that my project is going to be late because I can’t get those 200 story points, those 200 pieces of work, done in what I think it needs to be done.
Fletcher Hearns: That’s why, taking the metrics back here and using those as a one -day measurement are not good, but when you start to trend them out over time, you actually see that, “Oh, here’s what’s happening. Oh, this is good. I can now use this information to predict future performance of the project.” And I use one from a very agile. Velocity is only done on agile, and earned value which is traditionally done in traditional project management even though it can cross over to show how they’re all different but they’re all the same. We’re still using the same theories of if I find the right metrics, it’s going to show me what future performance will look like for my project.
Fletcher Hearns: Getting started. To get started with any measurement, what must you do? This is the one that when I walk in and start talking with people about this is the one that’s hardest for them to want to do. Does anybody want to guess at what the first thing you have to do to do accurate measurement? The answer is to set a baseline. I’m sorry. Please don’t hate me. That is, you have to set a baseline. You have to say this is what I think I’m going to do so that I can measure against that. Why? Without a baseline, you have nothing to measure. Try running a race without knowing where to start or finish. Try losing 20 pounds without knowing your starting weight. If you don’t have a baseline, what are you measuring against? And then, if you have the baseline, it’s controlling the baseline to make sure if it moves and changes, it does that. But without that, it’s very hard to know what you’re measuring against.
Fletcher Hearns: I want to really conclude with, because we’re going to get into more of this the next time, performance measurement. Always remember that, it’s a Dilbert one, doing great metrics, doing great dashboarding doesn’t mean that what you think the answer to what should be done with your project is going to have any relevance in any place on what might actually be done with your project. Why? Because you’re going to give the numbers to them, and people are going to make decisions based on information, facts and stuff that you may not know about.
Fletcher Hearns: So, company, organizational [inaudible 00:48:01] will come into it because you may have this thing that’s looking great and they’re still going to kill your project because somebody new came in. I know that because, like I said, currently about 80% of my customer base is in the government space, and every four years there can be a change in management at the most senior level that will affect how the projects that we are working on are classified. For four years, they’re the child in there. [inaudible 00:48:28] Yes, we want to do that, we want to do that. And then, the next person comes in and goes, “No, I don’t really care about that one. I want these projects to take forefront.” So, there’s always that. Just be aware of that because you’re going to do all these work and then someone’s going to make a decision based on something you have no knowledge of.
Fletcher Hearns: So, I want to take all your questions now. I’ve left some time here at the end to do that, and then I want to invite you guys to come back next week when I’m going to be talking more and more about what good measurement and performance reporting look like, and start talking about dashboarding and what that all looks like. And, as I said, the third one will be actually looking at how you collect the data to build the dashboard based on what we talked about today, is finding the metrics you should and must manage or want to manage. So, are there any questions?
Kyle: Hey, Fletcher. We do have one question here. Just a reminder, anyone who has a question, feel free to chat that over. We’ll take that question and answer that now. The question came in from Amy. I know on part three we’ll be talking more about using the tool, especially Microsoft Project, but she is curious if you can use Jira for trends or anything like that.
Fletcher Hearns: It’s very hard. One of my clients is using Jira for their agile software development, and Jira is a wonderful agile software development management tool. It’s not really a great project management tool. You can get some of the metrics out of Jira, but you can’t report with them within Jira. You have to pull them out and pull them in to some other system to do the reporting.
Fletcher Hearns: In fact, we have several clients where we work with them to pull it out of Jira and either put it into Microsoft Project, and we put out a Microsoft Project because all their projects are reported in the Project. We pulled the data out and used it in things like Excel. And you pull the information out of Jira, put it into Excel to be able to report on, as I said, that velocity graph that I showed. I don’t know of any way inside of Jira to be able to show that. I can get what the velocity is for any point in time, but being able to show that in a quick graphical format is not something Jira is designed to do. It’s not a project management tool, it’s an agile software development environment for handling agile software development or agile project management but not for reporting. It was never designed for that.
Fletcher Hearns: Amy, it’s a great question. But, no, I don’t know how to do that. I’d pull it out and put it into something else for reporting. Like, suck it into project and have a report out of it, or suck it into Excel and have Excel create the dashboard to report.
Kyle: Okay. Thanks, Fletcher. I just had a question come in from Eric. He asked, “Do you consider the quality of a schedule, i.e. complete logic, complete critical path, et cetera as a leading indicator for project success?”
Fletcher Hearns: Absolutely, Eric. It’s one of the things that you must do. If you don’t have a well-defined schedule, your chances of project success are diminished before you ever start to actually work on the project. In fact, in earned value management, it’s one of the things that if you don’t do it, I can guarantee you, your metrics for earned value will not be there. If you don’t have a well-defined schedule predecessors, successors, resources loaded, work hours put in there and you just start to shoot from the hip, no, your chances of … I’m not saying your project can’t be successful, but it won’t be successful within the constraints that were probably laid out on day one.
Fletcher Hearns: So, yes, Eric, you have to have a … I’m talking this across agile and waterfall. In agile, it’s just done a little differently because normally you’d set up, okay, I’m going to do 14 iterations till MVP, minimum viable product. And you have a backlog of things. You have to track the same, but if you don’t know all that stuff and you’re not set up to do that, then yes, it’s still a problem even in agile. It just doesn’t rear its ugly head as much in agile as it does in a traditional project with strict earned value requirements. It will show up in any case, but in earned value, it’s almost a killer to your earned value if you don’t set up a very tight schedule before you really start to do the measurements.
Kyle: That does it for the questions that have come in. I see that you have your contact info on the screen, so anyone interested in reaching out to Fletcher. You can actually take a screenshot. If you click the camera icon, if you’re watching this live, click the camera icon and that’ll take a screenshot of what you see there so you can have that info available for later.
Kyle: Yeah, that takes us pretty close to the end point here, Fletcher. Anything else before we wrap up today?
Fletcher Hearns: I want to thank Kyle and MPUG for inviting me to do this series. I’m really enjoying it. And, as Kyle said, if you want to, please feel free to reach out to me with any questions you might have, comments, anything I said blatantly wrong and then next week I’ll correct myself.
Kyle: Thanks, Fletcher. We appreciate it. Thanks for your time today and sharing with the MPUG Community. And yeah, just a reminder to everyone, this is a three-part course. Part two is in the chat box. If you click that, you can register for part two and save your seat there as well as part three, and that’ll be hosted in the next two consecutive weeks. Actually, I’ll get that info on the screen for you now. You should see that coming up. The activity ID for today’s session is on the screen to claim for the PDU. Here are those upcoming dates for part two on May 12th and part three on May 19th. Same time as usual for the MPUG Webinars. So, What to Measure and How to Report, and part three will cover Using Microsoft Project to Track and Report on Performance. We hope to see you there at those events.
Kyle: If you missed any of today’s session or would like to go back and review, the recording will be posted to mpug.com in just a couple hours. You’ll also receive an email with the link directly to that. That’ll be there for you.
Kyle: That does it for today. So, once again, thanks, Fletcher. Thanks to everyone that joined us live or is watching on demand. Thanks to those of you who submitted questions, we appreciate that. We hope you have a great rest of your day and we’ll see you back next week for part two of this training course. Thanks.
Fletcher Hearns: Thanks, Kyle.
Kyle: Thank you.