At our last CIO Breakfast Forum for 2008 (www.cioforum.co.za) in November we presented and discussed Agile Practice. We specifically looked at the use of the Scrum project management methodology and a practical implementation in an enterprise. The following review provides some context in terms of agile practice in project management and software development. There was a lively discussion and unfortunately we went over our usual time limit. This looks like a topic that may evolve into other conversations.

Tony Franco, a Certified Scrum Master (CSM) discussed and presented Scrum and highlighted some of the differences between the agile project approach and the more traditional waterfall life cycle based process of project management. He discussed the Scrum techniques and how organizations can overcome some of the constraints that impede communication and delivery. Improving project delivery is a typical concern and priority for Chief Information Officers (CIO) and the questions raised highlighted some of the areas of concern pertaining to adopting Scrum. We also discussed some of the experiences with regards to the adoption of Scrum and agile practice.

What is Agile About?

For those who have been exposed only to structured methods, corporate policies and procedures perhaps one can view the agile practice as a more natural or postmodern way of working. Given the large percentage of information technology (IT) related projects and in particular software development projects that fail, it may not be surprising that a number of alternative approaches have been developed and evolved over time. When eXtreme Programming (XP) was introduced by Kent Beck, many programmers embraced the XP set of practices that contributed towards better team dynamics. XP was embraced by programmers as a more natural fit to the dynamics of software development. The Agile Manifesto has been supported by many professionals who have adopted the agile set of principles that among other things places people over process.

And fundamentally, that is what agile is about, people. While many corporate organizations strive for an agile enterprise and look to automation as the means to implementing strategy, a truly agile enterprise is dependent on people. Processes, efficiencies, automation and information-flow are important to large organizations but processes, documents and machines cannot "connect the dots" in the way people can. Responding to the external social environment of an enterprise or solving problems that cut across the internal silos of an organization requires human intelligence.

Scrum is a project management approach that supports the agile practice. It encapsulates the practical methods that are relevant to managing projects. While XP promoted agile to software developers, Scrum has introduced agile into the project management world and into the enterprise. Scrum "packages" the method of agile project management and places emphasis on aspects such as the time boxed 2 to 4 weeks iterations called "sprints", the daily 15 minute stand up meetings, the functional requirements encapsulated as "stories" and the white board of stories and tasks. Scrum is to the product owner and customer what XP is to software programmers. It exposes everything and improves just about every aspect of organizational work - team dynamics, communication, problem solving, estimation, collaboration, transparency, agility or responsiveness to change. As demanding as it seems, it drives the business-IT interactions because the customers and products owners can see the benefit.

But what is truly radical about Scrum and XP in an enterprise environment is the liberation of common sense. Often the negative impact of establishing structured rules around processes is that it creates unnatural walls that inhibit humans from addressing problems that range from simple to complex and systemic. The Scrum Master or Coach plays the role of a Project Manager and is responsible for among other things removing "impediments". If the scope of work needs to change today, the Scrum Master and the Team deal with it. There is no hang up about "Scope Creep". There is no rule that says you can't speak to someone else or have to go through specific lines of communication.

Policies, procedures and standards drive governance, quality and efficiency through processes. But values, principles and practices provide humans with the intrinsic tools to solve problems and work towards a common objective. The agile practice is about building strong teams and strengthening individuals. Kent Beck outlined a set of values, principles and practices that constituted XP. Many people focused on XP practices such pair-programming and working 40 hour weeks (as opposed to programmers sitting in front of their computers for 12 hours a day and then burning out). But XP values such as "courage" are as important to the agile team dynamics. If a programmer does not have the courage to speak out or to estimate according to what he really thinks it will take, he may often get bullied into doing work that in the end does not achieve the outcome that is expected.

How Does an Organization Start to Adopt Agile?

It is evident from the discussion that organizational change is important. Successful adoption means that the organization as a whole must be won over. This can be done through demonstrating the benefits and improvement in project delivery and quality. As with any radical change that is introduced into an enterprise, the initial stage can seem more like a war zone and cause disruptions. It often helps if there is an existing problem or crisis that needs to be addressed and where the environment is almost desperate to try anything. But exploiting this kind of situation to achieve the means as an end can only worsen the state of the enterprise and demoralize people even further. The right thing to do is to absorb the pain, take ownership, apply leadership and be accountable. Where others may avoid being seen as the scapegoat, take the risk and try and understand the problem.

A method for doing this is to conduct a quick survey and capture all the issues that stakeholders have, making sure this is captured and validated as representing the business concerns across the organization. Having a clear view of the "enemy" as a set of problems rather than political divides or people can help to unify everyone towards a clear and common objective towards overcoming these issues. Doing research on Scrum and agile methods, you may find that there are common patterns that can be applied but be careful not to try and find problems for a solution.

The initial change that others may criticize as “chaos” and the frustrations people have with any change can be overcome in the first few months through the benefits gained from closer interaction and communication. Within a matter of weeks, teams can become highly interactive and the visible activity in itself can motivate and re-vitalize the whole organization. It can be beneficial to offer Scrum training to business customers and product owners.

There are Certified Product Owner courses that help non-technical role players to plan, estimate and work with an agile team. In the process, you will have the core of change agents that will win the support of staff, middle and senior managers and executives in the organization. It should never be a forced adoption but an evolving and natural way of working. The dynamics, deliverables and outcomes will speak for themselves. And as the original set of issues are overcome one after the other and as projects are delivered with increasing improvements, communicate and celebrate these as a team.

The iterative daily and 2-4 week sprints will amplify success and failure, making them more visible and providing more frequent opportunities for positive and negative internal and external assessments. The values will help individuals transform themselves to be more constructive team members. When things are not going well, courage will equip them to take ownership and accountability. When teams achieve and are commended, respect teaches them to credit others and to celebrate by respectfully honoring all. This is infectious. Soon others in the organization want to be part of this because as humans we all want to know we are appreciated and are making a difference.

Can Agile Scale?

A question was raised about the criticism that agile does not scale. Adopting traditional project management methodologies such as PMBOK or PRINCE2 at a program and project portfolio management level and using Scrum to manage agile teams can work well. One way to address the issue of scaling agile projects is to split the project management roles and functions where the Portfolio Managers manage the program portfolio (macro project management) and the Scrum Masters manage the Scrum back log (micro project management).

Organizations need to plan, prioritize, monitor and review a set of programs, projects and sub projects in order to manage the enterprise. An effective steering committee at the highest level involves the CEO, CIO and the executives along with senior managers of the lines of business. It is important to separate the programs and projects at this level as a portfolio and distinct from the “Backlog” of stories that are introduced into the time-boxed sprints of the Scrum teams. Essentially, what the steering committee manages are the set of projects over a period – perhaps reviewed on a quarterly basis to an annual set of strategic and operational programs.

We looked at a practice or model where Portfolio Managers were assigned to manage projects at a macro-level within the lines of business. Each portfolio manager has a number of Scrum Masters as direct reports and in turn each Scrum Master has a team or manages one or two agile teams. Having more than one team to manage can impact the ability of the Scrum Master to deal with impediments that occur on a daily basis.

This structure allows the Portfolio Managers to focus on the outcomes that the deliverables of the Scrum backlogs are intended to achieve. The portfolio managers are responsible for the interdependencies between projects and a collective of Scrum teams known as Scrum of Scrums. By focusing on a quarterly or periodic set of project priorities, the steering committee keeps the entire organization focused on what needs to be achieved.

We also discussed how teams can grow and then split into concurrent sprints so that work can be tackled in parallel. While it is best for teams to be co-located, there are ways of dealing with situations where resources span different locations. Making use of video conferencing technology, instant messaging, electronic black boards, software process and development tools can help overcome this challenge.

Does Agile Require Specialized Roles?

Another question that was raised and discussed was that of the roles required for agile software development teams. An agile team is a self-sufficient team and should ideally consist not only of software developers but the key functional skills that are required to support software development and the internal processes of an organization.

Scrum and XP focuses on 3 primary roles: the Product Owner or Customer; the Scrum Master or Coach and the Team. These are the “pig” roles, which are dedicated full time to the team. There can also be a number of “chicken” roles that participate or are involved. We discussed the model of having portfolio managers and also of the team consisting of developers, business and system architects, quality assurance and testing as well as configuration librarians to ease deployment.

Agile practice requires an agile architecture approach. The ability to have daily software builds is essential. It may not be practical to have a daily build but having an architecture that can be iteratively and incrementally developed and the configuration managed on a daily basis to enable rapid deployment and release into the production environment without compromising the integrity of the environment is crucial.

A fully functional software development team does not only include object orientated software developers. Both user interface design and database programming have become specialized development domains that often require a specific set of skills and individual characteristics. Agile architectures enable the separation of these concerns. In web based internet application development using so-called Web 2.0 and rich internet applications, the separation of concerns can be facilitated by adopting an AJAX-SOA approach.

Both Asynchronous Java Script and XML (AJAX) and Service Orientated Architecture (SOA) have been increasingly adopted into the mainstream of software development. In large enterprises that still support mainframe, you will likely see the enterprise applications classified into – Mainframe, Client Server and AJAX-SOA. Agile architecture is a different conversation in itself. But it should be apparent that the growing complexity of information systems requires specialization and those agile teams should consist of the skills required to achieve the results expected of the team.

Finally, we consider the strategic and operational nature of agile practice. The strategic benefits are obvious if it enables organizations to implement the programs that will achieve their strategic objectives. But can the agile practice work for operational functions? We shared some experiences of agile in non-IT operational environments. But this could also be an interesting follow up discussion on agile in IT operations and service management. We look forward to further conversations on topics related to agile practice.