Sunday, May 18, 2014

Detect Code Duplicates Using ConQAT

Finally, I had sometime to do this simple and invaluable tutorial on your first steps with ConQAT to detect code clones.
  • Open eclipse with conqat enabled
  • From the file menu - New - Other - ConQAT - Run Config:



  •  Select a parent project to create the conqat config. This may be any project or folder, even an empty project. 


  • Fill the missing parameters, then click the launch config link as in the next screen shot: 

  • After the run completes, and xml file containing all the clones are generated in the output folder. To view the clones, , goto clone detection - open clone file. 

  • Here you are:

  • Don't forget to open the clones perspective. This will help you visualize the clones.

Sunday, April 27, 2014

Diagrams of Effects

Diagrams of Effects were first introduced in the famous four-volume series: Quality Software Management by Gerald Weinberg. It is a great enabler when trying to understand the dynamics of a system exhibiting non-linear behavior, very similar to software development team systems.

Diagrams of Effects are similar to Causal Loop Diagrams (CLD), but it is slightly different in notation and is more powerful in modeling human interventions in the system. A diagram of effects consists primarily of nodes and arrows. Each node corresponds to a measurable quantity. Easy arrow corresponds to an effect (whether positive or negative) which the source node exhibit on the target node. This is a description of the diagram elements:

Element
Notation
Description
Node
A measured quantity. It can be actual measurement, or deduced from observation or by experience
Cloud Node
A measurable quantity, although actual measurement may not be done. Or, it may represent a conceptual measurement (like stress, pressure, etc.) which may be too expensive to measure, or just not worth the trouble.
Natural Positive Effect
If A moves in one direction, B moves in the same direction. This effect is natural (without human intervention)
Natural Negative Effect

If A moves in one direction, B moves in the opposite direction. This effect is natural (without human intervention)
Positive Human Intervention
Indicates a human intervention which makes the affected measurement move in the same direction as the movement of the cause.
Negative Human Intervention
Indicates a human intervention which makes the affected measurement move in the opposite direction as the movement of the cause.
Human intervention with open choice of Effect
Indicates a human intervention which makes the affected measurement move either in the same direction or the opposite direction, depending on the intervention.

Reinforcing & Balancing Loops:

What is interesting about Diagrams of Effects is that they reveal Reinforcing loops and Balancing loops. Reinforcing loops are cycles of effects which reinforce each other and may drive the system to failure. In the following example, more bugs lead to higher levels of stress which leads to more bugs. Also, it works in the other way as well, less bugs leads to less stress which leads to less bugs:


Another type of feedback loops is the Balancing loop, in which one effect balances another. In the following example, in case bugs increase, team exerts more effort in code review which leads to less bugs, which balances the effect of increasing bugs:
Such dynamic is not possible to be studied using  2-dimentional graphs. This is why Diagrams of Effects is an excellent tool for analyzing complex systems.

Saturday, March 29, 2014

Why change? Is your team aware of the value?

As change agents, we introduce changes in organizations we work for. Sometimes, these changes are integrated smoothly into the way the team works. In other cases, we find a lot of friction and resistance.

One of the things which facilitates change and alleviates friction is explaining the value of change for your team. Although this seems straight forward, but we usually do not exert enough effort to do that!

In one case, I didn't need to explain the value of change, although the change looked so naive and has no value. See what happened when I asked a crowd of 220 audience to change places, and they just did it with no single objection!!




This was part of my keynote last August at AgileAfrica 2013. I understand why would someone just follow a keynote speaker telling him to do something even he/she didn't know its value. If I were sitting at their place I would most probably think that there is value, and I would rather wait till I see it later on. 

I have done this experiment with smaller groups. The result was completely the opposite. They challenged this request (to change places). They felt skeptical about the value of this change. It was necessary to convince then with the value of change before they do it. 

Take care when you are leading change in organizations: In order to take the organization one step forward, you would better explain the value of change for the whole organization, or at least for the whole stakeholders who are affected by this change. 

Tuesday, March 18, 2014

Useful Statistics about Maintainability

Here are some useful statistics collected from several papers:

  • Approximately, 70% of software programs are identifiers. Therefore, identifiers names are of paramount importance for readability
Unused Features: From this paper:
  • Jim Johnson (the Standish Group) in his keynote at XP2012 reports that 45% of the features in the analyzed systems were never used
  • Industrial business information system showed that 28% of its features were never used
  • 25% of all method genealogies were never used
Code Clones and defects: Some very useful statistics from this paper: Do code clones matter?:
  • 52% of code clones are inconsistent
  • of these inconsistencies, 28% are introduced unintentionally, and 18% are faults in the system
  • Overall, 10% of code clones are defective
  • 9%-19% conformance of implementation to architecture, 72%-90% of these non-conformances are due to flaws in documentation
  • 10%-28% decay in architecture
  • 20% effort increase in maintenance due to code clones

Sunday, March 16, 2014

Duplication in Model Elements

Interesting idea. If you are using Model-Driven development, how would you know that you have duplicates in your models? are you aware that this may be a source of many issues? If yes, are you aware that you have duplicates? If yes, are you doing anything about it?

I came across a paper talking about this: Clone Detection in Automotive Model-Based Development. The author has implemented an algorithm to detect clones in models. It is interesting that the results of the study shows that 37% of the model elements duplicated at least once!

Model clones highlighted in red and light blue


Saturday, March 15, 2014

My Talk at Agile2013

After several months participating in Agile2013 with an experience report about refactoring legacy code, I find it very useful to write down things that I have learned, and my plans to benefit from this experience.

My talk at Nashville was very good al7amdulillah. the audience were very interested in the experiences that we've gone through. The first thing I've learned is that when you deliver a talk, you get better understanding about what you are talking about! There is a tradition which says: "المال تنقصه النفقة، والعلم يزكو بالإنفاق", or "expending money would decrease it, but expending knowledge would only increase it!"

The second thing I found is that teams really need a way out of the mazes of poor legacy code. I got two verbal and one written comment telling me that they are having so many problems with their code, and they felt that this roadmap of refactoring would be very applicable in their case. This was an indicator that I should continue working on more experiments and enhancing the roadmap more and more.

The results of the session evaluation was as follows:

Attendees: 50
  • 40 green (would recommend to other)
  • 6 yellow
  • 4 red
I had some interesting written testimonials as well:
  • "Amazing story of ingenuity and success! Excellent presentation"
  • "Very interesting talk. It gives me hope we can tackle our own mountain"
  • "Very good. Learned something we can apply"

One of the attendees gave me an interesting note that she missed half of the talk because of my english accent. This was an excellent feedback. In later sessions, I made a disclaimer that I'm talking "Egyptian English", which is similar to English but you need to concentrate more to translate as you hear :)

Another note is that data makes a difference with some audience. It just ticks with them when they see data. In a later talk about process increments, I got similar notes from other attendees. What I felt is that there are some types of attendees whom they start believe in what you say once they see some data!

Next Steps:
  1. I should continue the ongoing experiments and collect more data to correlate refactoring with business metrics to indicate real return on investment
  2. Conduct more experiments to collect more data. 
  3. Package the roadmap (along with rules or the game, the guidelines, tips and tricks, notes about tools), and give it a name.
  4. Enhance the tools, really don't know how, but I'm sure this will emerge in time. 

For your reference. This is the PowerPoint presentation of my talk, and this is the full published paper.

Saturday, February 8, 2014

Trying to Understand Why Kanban Works

Recently, I came across a book chapter explaining Little's Law; published by Massachusetts Institute of Technology. This chapter is an excellent read which is simple and yet digs into the heart of Little's Law. Reading about queuing theory is provocative and makes me relate many of the already established ideas to Lean software and Kanban.

One of the issues which I found this book chapter explaining very well is the difference between Little's law in its original formulation (which considers the Arrival Rate as one of the formula parameters) and its application in manufacturing systems (which replaces arrival rate with throughput). Here is an explanation:

Little’s Law
Little's Law state that:
L = λ W
where
L = average number of items in the queueing system
λ = arrival rate of new items to the system
W = average wait time for an item in the system
John Little argues that this law is robust and generic and holds exactly to any queuing system given an essential condition; which is "to have a finite window of observation that starts [when the system is empty] and stops when the system is empty" (p.88). Moreover, Little provides a mathematical proof for the law along with many applications in reality.

Little’s Law in Software Systems
Let's jump into software and map these concepts to Kanban. In Kanban, this is how Little's Law is stated:
WIP = Th * CT
where
WIP (average Work In Process) = average amount of unfinished items (bugs, user stories, change requests, etc.) in the development system
Th (Throughput) = Team output in unit of time
CT (Cycle Time) = Average time it takes the team to finish one item
Two basic differences between the two formulas: The first one talks about input or arrival rate, whereas the latter talks about output rate or throughput. The second issue is the condition which Little stated: that the system should start with 0 items and end at 0 items. In software, we rarely witness a system which has no maintenance requests.

To resolve these differences, Little indicated a more subtle condition of the Law to hold, which is no items to go inside the system and get lost or do not get finished, which he calls “conservation of flow” (p. 93). If we apply this condition to the system, we can easily show that the input rate = output rate, and therefore, we can relate Lambda (λ) with throughout (th).

For the second condition (system should start and end in 0 items), Little argues that the law “still applies, at least as an approximation, as long as we select a time interval that is long enough”. (p. 93)

What’s useful about this Law?
Little’s Law is a very powerful tool to use for managers. If we have two parts, and we can instantly calculate the third.

Increase team throughput
If your capabilities are fixed and want to increase throughput, the only solution is to increase WIP, and increase the team size. If this is not doable, and your input rate is high and getting higher, you may decide to decrease the cycle time instead of increasing the WIP and this may involve automating the deployment process.

Stabilize Cycle time
If you are struggling to stabilize the process so that to meet some SLA, Little’s Law gives insight about what to change in the system. You may either increase/decrease with WIP or empower your team to meet a specific SLA.

Study behavior of specific request types
Little’s Law works for a sample of item in the system as well. You may zoom into the items in the system and study the dynamics of a specific segment (like high severity bugs, for instance).

Monday, February 3, 2014

Interesting quotes

Quotes that deeply influenced how I think!

“I hear, and I forget
I see, and I remember
I do, and I understand”
-- Ancient Chinese Proverb

“Leadership is the process of creating an environment in which people become empowered”
-- Gerald M. Weinberg

"When projects fail, it's rarely technical"
-- Jim Johnson, the Standish Group

Parkinson’s Law:
“Work expands so that to fill the time available for its completion"

"An approximate answer to the right problem is worth a great deal more time than a precise answer to the wrong problem" 
-- John W. Tukey

“Insanity is doing the same thing, over and over again, but expecting different results.”
-- Albert Einstein

“Two things are infinite: the universe and human stupidity; and I'm not sure about the universe.”
-- Albert Einstein

“If you can't explain it to a six year old, you don't understand it yourself.”
-- Albert Einstein

“Anyone who has never made a mistake has never tried anything new.”
-- Albert Einstein

“A clever person solves a problem. A wise person avoids it.”
-- Albert Einstein

“Everything must be made as simple as possible. But not simpler.”
-- Albert Einstein

Thursday, January 30, 2014

Redmine Arabic

A while ago, I worked with a client who is using Redmine Arabic interface. I found the arabic translation of Redmine very weak. It is missing many translations and miss-translates many others and causes a lot of confusion.

The good news is that I have reviewed Redmine arabization and adjusted the arabic translation to match the majority of arabic speakers in Egypt, the gulf area, and the Middle East. Also, I have fixed many incorrect translations and added missing ones.

The new translation is released with Redmine 2.4.0, and here is the patch that I have submitted.

Monday, July 29, 2013

It’s none of the Managers’ Business!

A team is busy refactoring a very large and overly complex several hundred thousands lines of code. This team would hang on the wall this sign and instructions:


We are busy refactoring the product code for the benefit of all of us

It is a highly technical stuff which you will not grasp, even if you tried hard

We need to concentrate. Please do not keep nagging for status and end dates

When we finish, we'll let you know

Thank you for your understanding :)

Yes, this is a joke, but we've met teams doing this spontaneously! Here is the full story:

In a recent review of three companies which tried to refactor large products (hundreds of thousands line of code), we found that decisions were taken in all three companies to suspend the refactoring effort. Two of them decided to go for a full rewrite and the third decided to minimize further development in this particular product.

This appears to be impatient managers' decision. But, analysing the three cases we reached some very interesting reasons behind these costly decisions. The number one reason was the proposition by the technical team that: "it's none of the managers' business"!

Technical teams had this attitude. Managers had almost no involvement in planning refactoring effort, and the technical teams did not spend any effort to involve busy managers or report status to them. This attitude created opposition to further expenditure on refactoring and was one of the main reasons for management wanting to either discontinue the refactoring effort or rewrite the application from scratch.

We reached the following corollary about  how managers manage refactoring effort:
Managers will not sponsor any activity unless they can track and control it!
 Other successful refactoring attempts witnessed more support from managers. The basic difference was that managers used to receive monthly reports about the status of refactoring, and would accordingly decide on whether to accelerate or slow down refactoring.