It’s none of the Managers’ Business!

A team is busy refactoring a very large and overly complex several hundred thousands lines of code. This team would hang on the wall this sign and instructions:

We are busy refactoring the product code for the benefit of all of us

It is a highly technical stuff which you will not grasp, even if you tried hard

We need to concentrate. Please do not keep nagging for status and end dates

When we finish, we'll let you know

Thank you for your understanding :)

Yes, this is a joke, but we've met teams doing this spontaneously! Here is the full story:

In a recent review of three companies which tried to refactor large products (hundreds of thousands line of code), we found that decisions were taken in all three companies to suspend the refactoring effort. Two of them decided to go for a full rewrite and the third decided to minimize further development in this particular product.

This appears to be impatient managers' decision. But, analysing the three cases we reached some very interesting reasons behind these costly decisions. The number one reason was the proposition by the technical team that: "it's none of the managers' business"!

Technical teams had this attitude. Managers had almost no involvement in planning refactoring effort, and the technical teams did not spend any effort to involve busy managers or report status to them. This attitude created opposition to further expenditure on refactoring and was one of the main reasons for management wanting to either discontinue the refactoring effort or rewrite the application from scratch.

We reached the following corollary about  how managers manage refactoring effort:

Managers will not sponsor any activity unless they can track and control it!

 Other successful refactoring attempts witnessed more support from managers. The basic difference was that managers used to receive monthly reports about the status of refactoring, and would accordingly decide on whether to accelerate or slow down refactoring.

Well, Agile Teams Should be Appraised as a Team ... Hmmmm

It may be useful to think this way. Agile teams are cross functional, self organizing, and collectively accountable for work results. Therefore, they should receive one score at team appraisal, and hence one raise, and one bonus, and one etc.

Very true. But, appraisals have another merit other than calculating the yearly raise. It should be used an invaluable individual motivator for self improvement, specially for teams whose work products involve highly innovative and creative work .. wow!

Usually, what I have seen in enterprises is that individual appraisals were a source of all kinds of nuisances, for the individual, for his manager, for senior management, for all the organization. It is rarely accepted, and usually results in demotivation of 90% of the individuals who see the results as unfair and biased.

There are causes for such phenomena, and here are five of them:

1. Using process data in evaluation, like the number of bugs, or the code review issues, or the produced LOC per hour!
2. Using everyday events and situational happenings as evidence for good or bad evaluation
3. Mixing individual productivity and team productivity metrics, and using both interchangeably while evaluating the team.
4. Single-rater evaluation by direct manager.
5. Using the appraisal as a motivational tool. That is, let's appraise the team, and give them a raise so that they become motivated. This doesn't work, and the higher the raise become, the more demanding the team will be.

Ok, how to appraise Agile teams? I'll prepare a model for that. This model sets two objectives for evaluation: Evaluating the process, and evaluating the individuals. The metrics and techniques used to evaluate the process are completely different and disjoint from metrics and techniques to evaluate individuals.

This will be my work for the upcoming several weeks. Keep alert!

Detect Architectural Violations Using ConQAT

In the previous blog on how to divide code into component, I have outlined simple steps. One of them is to detect and count violations (invalid calls from one component to another).

The following are simple steps to detect such violations using ConQAT:

1. Add a new Architecture Analysis. Go to Project>New>Other 
   
2. Select ConQat>Architecture Specification: 
   
3. A new architecture specification file is opened. Start adding components and assign classes, packages or namespaces to it: 
   
4. Following the steps in the next screen shot to (1) Add component (2) Click on the component to open the properties window (3) Press the ‘New inclusion button’ (4) Write a pattern indicating the class or package name: 
   
5. Now, we should run the architecture analyzer to verify the dependencies and catch any violations. To do that, we will create a Run configuration. Open “Project>New> Run Config": 
   
6. Open “blocklib” you will find a long list of analysis blocks. Select the following: 
• For .Net code: “ILArchitectureAnalysisChain” 
• For java code: “JavaArchitectureAnalysis” 
  
7. Add the required parameters as in the next screen shot: 
   
There are some other useful properties like:

• exclude-dependency: to exclude the specified namespace or class pattern from analysis
• exclude-file: to exclude the specified file names from analysis
  

8. Press the ‘Launch ConQAT analysis’ link 
9. Go to the architecture specification file and press the ‘Switch Editor Mode’ button as in the screenshot below: 
   
10. Browse for the ‘architecture-assessment.xml’ file under the output directory and select it. The result of the architecture analysis is displayed as in the following screenshot: 
    

Some Tips and Tricks

• The ‘orphans view’ lists all classes which are not assigned to any component. Review this list and make sure that it does not include any classes which may impact your analysis

• You may use the ‘orphans view’ to incrementally build you architecture, as such:
1. Create all you components and keep them empty
2. Run the architecture analysis, you will find that all classes are orphans. 
3. Drag and drop classes or complete packages on to the component
4. If the package is not part of your code and need to be excluded, right click on the package and ‘Copy pattern’: 
   
5. Then, you may exclude this pattern using the exclude-dependency property

• You may right click the violations and take a decision to allow them, if appropriate:
  

A Simple Method for Dividing Code into Components

In the 3Q's method, the second stage of refactoring legacy code is 'Divide and Conquer'. In this stage, we discover and separate (or simply reorganize) the code into manageable components. Usually, this is done as per the following simple workflow:

What to do while dividing your code into components

This is rather a straightforward logical sequence of modularizing legacy code. What we are interested now is how each step is done.

In an experiment that I have done with one team, we have separated the classes into components and After the first step, this how it looked like :)

Analysis of violations between the newly specified components

Red lines indicate communication between components which should not be allowed. As you see, nearly there are violations between all components! This is very normal, because these components are specified for the first time; and previously, all classes were allowed to talk to all other classes in the application code.

But, how did we detect these violations? In a later blog: Detect Architectural Violations Using ConQAT answers this question.

I'm Presenting At the Agile Conference 2013

I'm presenting at Agile2013 inshaAllah :)

What I will present is the result of last year's work on continuous refactoring of legacy applications. Here is my session proposal details: Refactoring as a Lifeline: Lessons Learned from Refactoring.

The Agile conference is the biggest international conference about Agile Software development featuring more than 200 sessions delivered in 5 days. 

Detect Exact and Similar Code Duplicates Using ConQat

[Update: this post is related to ConQAT version 2011.9. However, the conQat blocks provided in this post will not work on the latest version which is 2014.1. Dr. Hummel (one of ConQat developers) has kindly shared with us the set of blocks for the newer version here. As per Dr. Hummel, the equality and similarity properties (described below) will be part of all future releases]

Part of my work on refactoring legacy code was to find a way to identify code clones (duplicates) in large data repositories; and during my experiments on refactoring, I found that it would be much faster to follow this order:

1. Remove dead code (CSRS ~ 600 LPH, also reduces 10% of duplicate code)
2. Remove exact clones (CSRS ~ 300-400 LPH)
3. Remove similar clones 

Where:
CSRS is the Cost Size Reduction Speed, measured in LOC per Hour (LPH)
Exact clones: are clones which are typically the same (probably they may differ in white space only)
Similar clones: are exact ones but with renamed variables.

The best tool I found was ConQAT, an excellent open source and very flexible tool. However, I had to adjust it a bit to enable detecting exact and similar clones.

Detecting Exact and Similar Code Duplicates:

If you would like to try this, download this adjusted set of blocks, then follow these steps to replace the old ConQat blocks on your machine.

1. Unzip the file “ConQat blocks - vX.Y.rar”, and replace the following folder: <conqat eclipse folder>\ conqat-2011.9\conqat\bundles\org.conqat.engine.code_clones\blocks
2. Press the “Enforce Full Model Rebuild”:
3. Open ConQat Runtime view and press “New”. You will find new parameters called “equality” and “similarity”: 
4. Add the “equality” parameter, and set the equality threshold to 1 (meaning that the clones should be 100% identical): 

See the Result!

This is an example of the result, only exact clones are listed:

Works for Which Language?

This applies on (and has been tested against):

• Java code using the JavaCloneAnalysis block
• .Net code using the CsCloneAnalysis block
• C/C++ code using the StatementCloneAnalysis block. Actually, the StatementCloneAnalysis may be applied on 10 different languages. Review the ConQat documentation for more information. 

Enjoy!

The First Retrospective

Being there while the team is doing their first retrospective is an invaluable and full of emotions experience. You notice the team members know exactly, for the first time, what they are doing: What they sought to do, what they did well, and what went bad. More important, you see the team get energetic, prepared, and all set for the next sprint, sometimes can't wait to rectify the side effects of the first one.

It is an experiment that I'm doing every now and then with new teams doing agile adoption. I see that it is important to study the first retrospective more deeply. Surely, it will give us insight about team dynamics and good practices of agile adoption.