3Q's Method: Measuring the Progress of Code Refactoring

Currently, I'm assisting in several projects to refactor large code bases of legacy applications. These are products of very bad, or better say, deteriorated design, and other products which did not follow any design principles what so ever.

First of all, let's agree on the basic idea that you cannot manage what you cannot measure. So, how would we measure how good or bad the code design?

The strategy that I will employ is the 3Q's strategy, described in the below diagram:

Systematic refactoring of legacy code using the 3Q's stratey

The first Q: Quick Wins

The first stage is to catch low hanging fruits, like identifying and removing dead code, removing duplicate code, and reduce method length. At this stage the following measures would be helpful:

• Cyclomatic complexity: should be 10 or below
• Average method length: 15 or below
• Code duplication for 3 (or greater) lines of code
• Overall code size: should be monitored and set targets for reducing it

The Second Q: Divide & Conquer

The next stage is to start thinking of pulling apart components, whether business (or functional) components or utility components. This re-organization of code is essential to break the complexity of code and will open a large list of refactoring opportunities. At this stage, we will add the following two measures:

• Instability: to measure correct use of design layers
• Efferent and afferent coupling between component interfaces and the outer world

The Third Q: Build Quality In

The last stage is to start writing unit tests to test component interfaces. This is necessary so that to baseline the code quality and start doing more profound refactorings. At this stage, we will add the following measure:

• Unit tests code coverage

Development Process Effectiveness and Efficiency Measures

All of the above are measures of "good & healthy code". However, how would I measure the improvement of the development process itself? in other word, how would I know whether or not these measures improved the development process effectiveness and efficiency? The following measures would serve:

• Ripple effect (aka change impact): Number of touched methods per change. This measure should start high, then decrease over time
• Fault feedback ratio (FFR): Number of injected bugs over number of resolved bugs. In healthy projects, this measure should be less than 0.3
• Average defect density: Number of defects per code size unit, averaged for all changes in an iteration. This measures the amount of defects, whereas FFR measures the healthiness of the code fixing process
• Average cost of one change per size unit: This is a bit tough to measure. But, depending on the nature of the product, changes can be sized and the cost can be normalized by the change size

It is worth mentioning that we should record readings for the development process measures starting from day 1. This would be the only evidence for improvement to higher management. It would be very indicative to tell the senior management that the FFR has decreased from 0.7 to 0.34, rather than telling them that the overall code size decreased from 35 KLOC to 20 KLOC :)

If you have previous experience with similar projects, which measures did you use?

Visio Activity Diagrams Stencil: A Valuable Tool for Lean Analysis of Your Process

UML Activity diagrams is an excellent tool for process modeling, and I have been using it for about 9 years now. One of the great uses of it is to model your current process, and visualize it to discover non-value added activities, or waste in other words.

This is a simple model for a typical Srum process, modeled in UML activity diagram:

What I have to you is a visio 2007 stencil for drawing such beautiful activity diagrams!

to download the stencil, follow this link: UML 2.2 -Activity Diagrams.vst

Agile Configuration Management (2): Towards a Practical Definition of Software Configuration Management

Although definition should be simplifying the concept, in case of Software Configuration Management, the definition is complicating the concept! Take, for example, this definition:

"A discipline applying technical and administrative direction and surveillance to (1) identify and document the functional and physical characteristics of a configuration item, (2) control changes to those characteristics, (3) record and report change processing and implementation status, and (4) verify compliance with specified requirements" - SEI CMMI Glossary

To complement this definition, SEI added references to 7 other definitions: configuration audit, configuration control, configuration identification, configuration status accounting, configuration item, product, and audit. What this effectively does is adding to the complexity of the definition!

On the other hand, there are some other definitions which are simple and to the point, and in the same time give a clear explanation of what Configuration Management means. It may not receive a unanimity among theorists that it is correct. However, in itself, it proposes a clear definition of CM, and I personally believe that they are excellent definitions. These are two definitions:

"Software CM is a discipline for managing the evolution of computer program products, both during the initial stages of development and during all stages of maintenance" - ANSI/IEEE standard 1042-1987 (withdrawn standard)

"In software engineering, software configuration management (SCM) is the task of tracking and controlling changes in the software" - Wikipedia 

These two later definitions captures in simple terms the essence of Software configuration management as per its original intent. Building on this definition, I have added a 'Capability-oriented' definition, which defines SCM in terms of capabilities it adds to the team:

"Software configuration management enables the team to trace releases, work-items, and work products to each other"

A strong SCM environment empowers the team to relate workitems (what has been done) to work products (artifacts of work done) to releases (packaged and delivered software products). This is what I describe as a 'Strong configuration management environment'.

Implications of this definition is huge. It means that the team may instantly know the history of a workitem (say  a bug), when it was released and which artifacts or code changed due to it. The team may also know every thing about a specific release to a customer, which bugs or user stories were included, and what code files or documents delivered as part of this release.

Agile Configuration Management (1): Does it Make Any Sense?

Lean thinking is one of the pillars of Agile software development. This title: "Lean Configuration Management for Agile Teams" is the latest workshop I'm conducting at SECC. Now, I'm giving my self an opportunity to write, why this topic is important.

Configuration Management is one of the great successes of Software Engineering. It was marked by great persons like Gerald Weinberg as one of the achievements of software engineering is the 90's. However, the older the topic, the heavier it became. It is currently perceived that a middle size company would need about 7-10 templates, 4-5 procedures, and many sub-activities to implement a "good" configuration management environment.

In this workshop, I tried to dig into the essence of CM, and what is really useful about it. I have gone through texts dealing with this topic, and reviewed all the previous implementations I have gone through. I tried to bridge a link between theory and practice. I was doing this in order to answer the question of one of my colleagues asking: "All of this stuff have absolutely no value, why are we doing this?". I was also trying to answer another question about how to become a CMMI-L3 company, while still Agile and Lean, specially with regard to CM process area.

Actually, many of the practices which is currently implemented as part of the CM process adds no or little value, or may add value is other contexts, other than software development. Also, I have seen many practices or real value, but implemented in a completely incorrect manner, which made it of no value for this specific company or team.

It is time to implement "lean" configuration management, which achieves the utmost benefit for the team, with the minimal waste or overhead. keep watching my next posts.

I'm presenting at the Agile Conference 2011

I'm presenting at the Aile Conference 2011, next August, in Salt Lake City, Utah.

This is an interesting experience for me, as it is the first paper, and the first conference participation. It happened to be the best Agile conference in the world.

My paper is titled: "Process Increments: An Agile Approach to Software Process Improvement". You can browse the abstract and the conference proposal at this link.

See you there inshaAllah!

Motivating teams to pilot Agile projects

This is an excellent advice by Gerald Weinberg, in this interview. The idea is that when doing organizational change, we need some projects to pilot changes before applying then to the rest of the organization. If projects are selected by management, the project team will view it as an assignment and may not cooperate very well. This is a problem that we need to handle when working with Agile adoption projects.

Weinberg suggests an excellent technique that he used many times, and had excellent results. He builds upon the innate developers' attraction to new ideas, technologies, frameworks, etc. The technique is to announces that if any team wants to become a pilot, they have to defend why they deserve it!

Simple and effective. He says that teams argued that instead of having one or two pilots, they all want equal opportunity of becoming pilots. Far beyond what a consultant would dream of!

Patterns of Agile Adoption Failure

Many teams are trying Agile methods, or rather techniques. Many of them fail, and suffer from this failure, usually because the failure induces resistance in the rest of the organization, and makes later attempts to Agile adoption much more difficult.

The following are four patterns of immature Agile implementations, where team concentrate on some superficial agile techniques without really embracing Agile values and principles:

• Usually team starts with stand-up meetings, and it fails, because it simply hurts their legs for standing 1-2 hours daily! Moreover, the meetings are boring, and take a lot of time discussing issues. The team feels that this "Agile" thing is naive and wastes their time.
• Teams do sprints:
1. Either one complete sprint for every waterfall phase. That is a sprint for analysis, another for design, another for development, and a final one for testing. So, it simply becomes sprintfall rather than waterfall !
2. Or one sprint for every big requirement or group of requirements. The problem with this approach is that no feedback loop exists. In other words, they have split a big waterfall project into smaller ones. Just bigger management overheads
• Teams use sticky notes and decorate the walls with whatever they do, even with requirements in a typical waterfall project. After a while, they lose interest in updating the board notes. Because every step take a lot of time (2-3-weeks), and they cannot see any value of maintaining two databases of requirements and tasks (board and electronic)
• Team does everything just right, but do not take feedback from customers. Somehow better, but still waterfall in the large. Teams iterate to get feedback and to adjust their development effort along the way. If they do not get feedback, the value of iterating is not realized

These are some failure patterns, there are many others; if you have any experience with this, please let us know about it :)

Why have principles behind practices and techniques?

The answer is very simple. If you can identify the principle behind the practice, you will open the door for creativity to invent other practices which may better realize the principle!

Agile Project Management Tools Evaluation

Project management, configuration management, release management, change management, et cetera, are all areas of interest which need tool support. Without such a tool, the area of management would be simply non-existing. I have seem many instances of organizations which assume to have good management practices, all document, email, ms-project based. When I challenged them to do one simple traceability from one requirement to its changes, they either take some good amount of time, or they fail to answer.

Sometime ago, I did an extensive research on the net to look for good open source tools which does the job. Every tool I gone through, I did installed it, experimented with it, and looked deep in its features. I had limited my research to tools which fulfills these criteria (so you may find many other tools not included in this comparison, so most probably they do not fulfill one or more criteria below):

• The tool is open-source (not even share-ware)
• The tool has an active development team, which is committed to continue support the tool.

Here is a word document which lists the results of the research. Please note that information is this report is not updated, and will not be updated. So, I recommend that you continue your own research because you may come across a new tool which I might not come across.

Hope this effort helps you in your new projects.

Subversion folder structure

One problem I faced is how to organize the svn folder structure, so that it accounts for documentation which is not part of any baseline.

I have solved it as follows:

> Project 1
  > Trunk
  > Branches
  > Tags
  > Archive

If you expand this structure, you will find something like this:

> Project 1
  > Trunc
    > Management & Analysis
    > Design
    > Testing
    > Code
  > Branches
    > 4.2
    > 4.3
    > 5.0
      > Management & Analysis
      > Design
      > Testing
      > Code
  > Tags
    > 4.2.0
    > 4.2.1
    > 4.2.2
    > 4.3.0
    > 5.0.0
    > 5.0.1
      > Management & Analysis
      > Design
      > Testing
      > Code
  > Archive
    > Meetings
    > Reports 

As you can see, what goes in the Trunc folder is only fully configured items (which are part of the baseline). Whatever else is stored in the Archive folder, which is one folder for the whole project, no need to put it under the trunk.

As I did above, I have added two folders under the Archive folder: Meetings & Reports. These are the two broad categories which should not be part of any baseline.