Lecture
A development department may include several dozen Scrum teams working together on shared code in the same branch of a version control system. This article describes the methods used to scale the Scrum approach and to manage inter-team dependencies.
In October 2006, salesforce.com's R&D organization began a major transition from a waterfall model to agile methodologies based on Scrum. At that point, 10 months had passed since the previous major release, and the release date of the new one had already been postponed five times. Many people were frustrated that the product was released rarely and with serious delays. We did not wait for the release to finish; we reorganized the existing teams into Scrum teams and, using Scrum processes, shipped the release in February 2007. Since then, using our new agile approach, we have shipped five more major releases (3-4 months long each) of our suite of SaaS applications and the Force.com platform. Each of them landed exactly on the planned day.
During the reorganization we followed Scrum's recommendations for individual teams, but paid little attention to interaction between teams. When forming teams, we tried to minimize dependencies between them, but the code did not change overnight, so quite a few interdependencies remained. Quite soon we introduced Scrum-of-Scrums meetings. These meetings helped us discuss problems and status, but meetings alone were not enough. While working on the last five releases, we tried and refined additional approaches that improve collaboration between teams. In the rest of this article we describe some of the difficulties of managing dependencies and how we overcame them.
Product development at salesforce.com consists of three divisions: Applications, Platform and Core Infrastructure. Together these three divisions contain more than 30 Scrum teams. Each Scrum team has developers, testers and a Product Owner, as well as a Scrum Master (usually a development or test manager, or a Program Manager). For complex features, Scrum teams also involve representatives of other functional teams, such as System Testing, Documentation, UI Design, Usability, Technology Operations and Release Engineering.
Typically all Scrum teams work on the same release in the same code branch. Our products are interconnected, so the features of different teams can be very tightly intertwined, both functionally and technically. From the technical implementation standpoint, many teams use shared code. From the functionality standpoint, teams working on features for the same end user must collaborate closely to create a harmonious and consistent user experience.
When so many teams work on one release, delivering shared features and changing shared code, it is impossible to foresee all the problems, surprises, changes, failures and successes that will be encountered along the way. This unpredictability is one of the reasons why managing dependencies is so hard. Later in this section we note a few more reasons. Any solutions must address these underlying conflicts and challenges.
Identifying dependencies is the first step toward managing them. Like many mature products, salesforce.com's systems are too complex for any one person or team to know and remember all the interconnections. To identify and understand dependencies, we increasingly have to rely on the collective knowledge of many people. It is critically important to pool this knowledge and to connect the people who have the information with the people who need it. Experts with a broad understanding of the system play a key role in determining dependencies and the consequences of proposed changes. However, as the system grows and becomes more complex, it is hard to avoid specialization and fragmentation of knowledge. Moreover, with the current volume of changes, it is impractical to ask experts to check the details of every new feature. We need a way to identify changes that are highly likely to affect (or depend on) the work of other teams, so that we can give such changes special attention. Of course, this is not so simple, because sometimes relatively small changes have a serious impact on the whole system.
Dependencies can lead to conflicts between teams. If one team wants another team to do something, the Product Owners of those teams discuss and determine the priority of that work. If the discussion results in agreement on timing, managing that dependency is fairly simple from then on. If no clear agreement is reached, or if the priority of the work is still low, the uncertainty and level of risk around that dependency increase.
Conflicts also arise when one team makes a change that adds work for another team. For example, when one team changes the architecture so that the others have to adapt to those changes. A change does not always bring obvious benefit to other teams, so people sometimes feel resentful and indignant about the extra work, especially when it appears suddenly. We try to identify such dependencies early, but some of them inevitably appear later, as a result of changes already made or after priorities are revised.
One of the advantages of Scrum is the ability to change a team's priorities for every sprint. However, this complicates work with dependencies. You cannot simply identify and think through all the dependencies at the very beginning of the release cycle. Analyzing interdependencies is more like a continuous, ongoing process. As work proceeds, dependencies appear and disappear while teams clarify the details of what needs to be done. Therefore the process of managing such dependencies must be dynamic and active.
Short release cycles leave less time for coordination between teams, so interdependencies and their impact must be identified early. At salesforce.com, release cycles overlap, in the sense that planning of a new release begins before the previous one is finished. This is intentional, because some teams are already free and can start work on the new release. However, other teams may lag behind with planning and skip the discussion of dependencies for the next release. This is a serious problem, given the importance of collective knowledge for clarifying interdependencies.
In this section we discuss the specific approaches we use to manage dependencies and to cope with the challenges listed above. The general strategies used in these approaches are:
About a month before the current major release ships, we hold a kickoff meeting for the next release. All employees of the company attend the meeting. The vice presidents of the business divisions describe in general terms which features each team plans to work on in the new release. This meeting noticeably helps manage interdependencies. First of all, it prompts teams to prepare their initial release plans by the target date. If a team still has unfinished work from the current release, that team starts to delay the planning of the next release; in this case the meeting helps get planning done on time. It is hard to identify dependencies and discuss commitments with a team that has no idea what it will be doing. The kickoff meeting serves as a synchronization point for teams and helps ensure a productive discussion of inter-team dependencies.
In addition, after the meeting the teams' release plans become common knowledge. We strive to achieve a high level of awareness of what all the teams are doing, because we believe this makes people think about and discuss interdependencies. All R&D managers take part in the kickoff meeting, which helps engage people and focus attention on release planning. A high level of participation helps align expectations and inform the board about release plans.
Of course, plans can change and do change as work progresses. Not long ago we began reviewing an updated kickoff presentation during the monthly sprint reviews. The updated presentation shows which features have been dropped or added, and greatly helps keep plans clear and public throughout the release.
Once initial release plans are prepared, teams can discuss interdependencies more concretely, and many such discussions arise spontaneously. In addition to this, we have found it very valuable to hold a more formal dependency identification session, where representatives of all teams work together. It is most useful to hold such a session shortly before the release kickoff meeting, to improve the quality of the plans announced at that meeting.
The dependency identification session is quite simple. Representatives (most often the Scrum Master or Product Owner) of all Scrum teams gather in a room with a large whiteboard. In the first part of the session, all participants simultaneously draw two diagrams of dependencies between teams on the board. One diagram shows the teams that need another team to do something for them. The second diagram shows the teams doing something that will affect the work of other teams. We use a few simple rules for these diagrams:
At first there is confusion while everyone draws their own interdependencies, and observers ask questions and point out additional dependencies. After about twenty minutes the diagram stabilizes and the comments dry up. After that we ask a representative of each team to briefly talk about their dependencies.
In a very short time we create a complete picture of the interdependencies in the release. Fig. 1 shows a typical result of a session after it has been cleaned up to be more readable. Hot spots (that is, teams with a large number of dependencies) are immediately identified. Teams with unagreed dependencies are also visible. Such teams are at the greatest risk of missing their plans until their interdependencies are agreed.
Fig. 1. Example of an inter-team dependency diagram
Recently, within a week after the kickoff meeting, we have been holding open meetings whose main purpose is to create a forum for discussing questions and problems related to the teams' release plans. We ask that at least one representative from each Scrum team attend these meetings; otherwise attendance is optional. Usually at the start of an open meeting the participants propose topics for discussion. Then, for 45 minutes, the topics are discussed in groups, and afterward the groups report the results of their discussion to everyone else. Most often participants want to learn the details of new and useful functionality, as well as about changes that will affect other teams. Participants in the open meetings noted that these discussions were instructive, and people also found it interesting to talk with colleagues from other teams with whom they do not interact in their regular work. Open meetings are still a new format for us, so we are trying different variations to understand what works best, for example:
The main goal of feature design review meetings is to raise the overall level of quality and usability of our products, by means of:
These cross-team meetings focus on the design of functionality and user interaction for features that we consider key, the most complex, or especially exposed to risk. At these meetings we strive to analyze and identify previously unnoticed inconsistencies and dependencies.
There are two main types of these meetings. Early in the release cycle, participants focus on design concepts and strive to achieve consistency and effective interaction between existing and new features. At this stage the meetings usually involve Product Owners and user interface (UI) designers, and occasionally representatives of other functional teams, such as development and QA.
Around the middle of the release cycle the composition of participants changes: now they are mostly QA representatives and, in part, development, and the discussions focus on implementation details. These meetings provide understanding and clarity regarding:
The Virtual Architecture Team (VAT) is "virtual" because it includes developers from all the Scrum teams. Team members work in the VAT without stepping away from their main duties in a Scrum team in the Applications, Platform or Core Infrastructure business divisions.
The VAT is responsible for maintaining and evolving the architecture of our software. The team defines the road map for architecture development, reviews major changes from an architecture standpoint, and defines coding standards to ensure code compatibility and maintainability.
The VAT manages the backlog of important architectural projects and necessary refactoring. As the product evolves, we have to redesign features and get rid of old, suboptimal code. Sometimes developers do not want to change programs that already work, even when they understand that the program's code leaves much to be desired. Product Owners prefer to see new features being developed rather than something that already works being reworked. To counter such attitudes, each of our business divisions is required to schedule at least 20% of the time in every release for changes recommended by the VAT.
The VAT meets twice a week for two hours to review the technical implementation of products and features being created by the Scrum teams. Teams working on the most complex features of the release are required to present them to the VAT. The VAT gives the Scrum team feedback on how their technical decisions will affect other teams, and which developments by other teams may affect the feature. The VAT focuses primarily on technical implementation, especially on aspects of scalability and performance. If a feature's design requires major changes, the Scrum team is required to present the feature again in the same release cycle and show the VAT how they changed their design.
Our web-based infrastructure for automated builds, testing, and triage provides continuous integration for the build system and lets us track the state of every line of code. This infrastructure is a key element that allows all developers and QA engineers to work with a shared code base.
The main test suite, built on a JUnit extension, makes it possible to create functional tests using the Force.com API. In addition, we have a UI testing framework that uses Selenium to automate test cases that must be run through the UI.
Here are a few fundamental principles that define our approach to automation:
A Scrum-of-Scrums meeting is held weekly in each business unit, and there is also a "Scrum-of-Scrum-of-Scrums". Sometimes we organize an additional Scrum-of-Scrums for a group of teams that work closely together toward a common goal. We have tried several different formats for the Scrum-of-Scrums. We started with the standard "4 questions" format, in which teams reported what they had done, what they were going to do, what was blocking them, and whether they were doing anything that affected other teams' work. At first this worked, but it soon became tiresome and boring because of the large number of teams. Then we moved to a more open, self-organizing format in which participants proposed topics for discussion by writing them on a board at the start of the meeting. This made participants take responsibility for the content of the meeting and led to more productive discussions. The question of cross-team dependencies is often raised during the Scrum-of-Scrums, especially in the early stages of development. A dependency identification session is held during the Scrum-of-Scrums, and over the following two to three weeks this meeting often discusses changes in cross-team dependencies.
When a team moves to Scrum, written reports often become unnecessary, since clarity about status is now provided by other things (sprint reviews, burndown charts, daily stand-up meetings). When we moved to Scrum, we decided to keep lightweight weekly status reports prepared by each Scrum Master. This seemed redundant while we were using the "4 questions" format for the Scrum-of-Scrums. However, with the current open Scrum-of-Scrums format, the weekly reports have become an important complement to that meeting. We do not discuss the status of each team during the Scrum-of-Scrums unless it is raised as a separate issue, but this status is always available in the weekly report. The report lists all dependencies, blockers, and risks. Of course, such reports require a little extra effort from Scrum Masters, but the clarity they provide justifies the cost of producing them.
We have shown that scaling Scrum is feasible, and we are confident that we can continue to scale it as the company grows. As the number of teams grows, managing dependencies and coordinating between teams becomes a difficult task. A reliable build and automated testing infrastructure is critically necessary. Raising awareness and synchronizing through the kickoff meeting, the dependency identification session, the Scrum-of-Scrums, and lightweight status reports helps a great deal. Ultimately this leads to effective communication, collaboration, and knowledge sharing between teams. Approaches such as a virtual architecture team, open discussions, and the Scrum-of-Scrums are intended to aid communication and support collaboration.
SCRUM
Software implementation (Agile)
Comments