One of the fascinating task my consulting brings is the opportunity to conduct in-depth studies of complex issues about knowledge sharing in a variety of organizations. This post is about a recent study I conducted for the Defense Intelligence Agency (DIA) to look at the Lessons Learned problems across the Intelligence Community (IC) and within DIA specifically. Although the study was conducted within an industry that has been publicly taken to task for its inability to learn from its own experience (The 9/11 Commission Report and the Commission on WMD in Iraq) the challenges these agencies face are not unlike those many other government agencies and corporations have to confront.
In this study I examine some of the causes of those difficulties and make recommendations about how the IC might make better use of what they learn from their own experience. With DIA’s permission I have excerpted both the challenges and some of the recommendations from the larger study.
Lessons Learned Challenges Across the IC
1. The individual nature of intelligence work
Unlike the teamwork that characterizes most work in both corporations and government agencies, intelligence work is still largely an activity that an individual analyst or collector conducts independently. For example, performance metrics tend to be based on numbers of intelligence reports from individuals. This view is changing slowly.
Although individual analysts may have suggestions for making changes in how they each get work done, they tend to not see value in coming together to look at the overall process. Simply executing a work process does not produce in-depth understanding about the system in which the process operates, or how to improve it. When personnel both work and are rewarded as individual contributors they perceive little value in joint reflection
2. Disconnect Between Gathering Lessons and Implementing Them
The most frequent concern expressed across the IC is the difficulty of getting lessons implemented.
There are two major factors that impact this gap. First, those responsible for producing lessons are seldom accountable for the implementation of what has been learned. The lessons learned function in most of the Intelligence Community is located in non-mission support units such as training, IT, or administration, which necessitates handing off the results to mission units for implementation. Under the best of circumstances this handoff of findings from studies and reports to ownership by operations is notoriously difficult to accomplish.
Secondly, most lessons learned efforts lack the initial identification of a specific target audience. There appears to be two general audiences for lessons learned across the IC: 1) senior management, who may be briefed or receive reports on the findings, with the intent that the findings should result in policy and procedure changes; and 2) a general audience, which is expected to “pull” lessons from a repository on an as-needed-basis to address a specific situation. Without a pre-designated mutually agreed upon target audience for the lessons, recommendations necessarily lack context that would otherwise make them actionable.
3. Negative Connotation
The term “lessons learned” carries several negative connotations in the minds of most IC personnel. First, the term is primarily associated with failed events or initiatives. Reports based on lessons learned studies are perceived as attempts to find out why the failure occurred and therefore, for many people, are associated with or seem similar to an Inspector General inquiry. Secondly, when lessons learned reports are developed in-house, those responsible for constructing the reports may have to deal with political fall out and attempts to suppress or modify the findings. Finally, there is a general view that lessons learned are of little value to accomplishing the mission and thus are usually just shelved or buried.
4. The Lack of Reflection
The op tempo inside the IC is high. Projects, teams, change initiatives, and strategic operations rarely pause to consider what they have learned from their actions. Without some level of group reflection and discussion, it is difficult for those involved to assess what is currently happening in order to correct course, or at project end in order to derive lessons learned.
Although content review procedures are in place for IC analytic assessments, few, if any, processes are in place to reflect upon the effectiveness of the analytic production process. Without such reflection the production methodology does not get updated nor can it benefit from best practices discovered during the production process. Unless reflection is valued and time for reflecting upon actions is built into the way analysts work, the op tempo will take precedence and IC organizations will lose performance improvement opportunities.
Process improvement requires deliberate reflection on the cause and effect of actions taken and results produced across the whole of the system. Coming together as a group to reflect on and discuss the total process increases the identification, validity, and usefulness of lessons by engaging a diversity of thought and experience.
5. The Retrospective Approach as the Predominant Approach
Lessons learned across the IC tend toward retrospective content, formatted either as composite in-depth reports or lessons from individuals placed in a repository. Retrospective lessons are valuable, particularly when they are developed in response to a question from a specific leadership group that intends to implement the findings through policy or procedures. However, there are many tactical and operational lessons that are more effectively shared and acted upon in real time. Intelligence operations, particularly at the tactical level, have a short shelf life, so that lessons that are over a few months old are no longer valid.
Recommendations resulting from the study (References to specific intelligence initiatives have been removed)
1. Embed reflection in workflow processes to enable organizational learning
Scheduling group time to think about how the work got done, reflecting upon what went well and what could have gone better, results in continuous updating and improvements to work processes.
Many government agencies and corporations that must increase the speed of their learning, do so by planning regular group reflection into their work processes and production schedules, e.g. NASA’s Pause and Learn (PAL), The US Army’s After Action Review (AAR), and Intel Corporation’s Knowledge Harvest.
Whatever reflection process an IC agency employs it should have the following characteristics:
• be simple enough to be implemented by its own employees with a minimum of training (NASA holds periodic workshops to train local personnel in the PAL technique)
• make use of a recognizable label (for example, AAR, Fast Learning) that is employed across DIA. A label elevates an action to the level of an approved practice as opposed to a one-off event. Once established, its value is known and accepted as a part of the way we continually improve our processes.
• utilize a standardized, repeatable format
• engage each person involved in the work process in the reflection process in order to bring diversity of insights and experiences. When reflection is limited to supervisors much of the experience at the tactical level is lost.
• conduct reflection sessions on a regular or scheduled basis, rather than only when things go wrong
• capture and understand what contributed to successful outcomes that should be retained
• focus on creating lessons that are within the group’s scope of responsibility to implement, rather than developing findings for those above them or for other groups to implement.
• reflect on what needs to be done differently next time rather than attempting to place blame
Reflection processes need to be built into the workflow process at all levels of the organization. Management as well as frontline workers accomplish work though processes that could benefit from periodic reflection. For example, every office periodically holds town halls. As a group, Office Chiefs could benefit from reflection on what works about town halls and what does not? What has been learned about the town hall process to make it more effective?
Periodic reflection on process is course correction rather than retrospection. It occurs not at the end of an event or initiative, but while the initiative is in progress – learning in real time
2. Empower groups at all levels to take action that implements the results of their own reflection
The culture of DIA is one in which people at lower levels expect to be asked to identify problems for upper management to solve, rather than finding solutions themselves. Although there are issues in any organization that are beyond the scope of a team or workgroup to resolve, the change that is needed is for groups to recognize they have the knowledge and the power to make improvements to their own work processes. That requires managers above them to support the reflection process that leads to new insights, and to acknowledge the ability of units to identify and implement useful revisions and propose new processes.
The shining exception to this cultural norm of passing all problems upward is DIA’s Crossing Boundaries, (see The Untownhall) where the expectation is that the idea provider is empowered to implement the solution. It is this spirit that needs to be infused now more widely through embedding reflection process in the production workflow.
3. Reduce the difficulties of hand-offs by involving those who can implement lessons learned findings
Hand-offs of lessons learned from those who gain and document its insights to those process owners with authority to make changes are a very common failure of lessons learned programs/initiatives/activities. Hand-offs occur when a team, made up of people external to those being studied, conduct a study and then make recommendations to those who are expected to make changes.
The likelihood that action will be taken based on the findings of a lessons learned study are greatly increased when those who are expected to implement the findings are involved in all phases of the process of creating the lessons learned - from the data collection (whether survey, interviews, review of historical documents or observation), through analysis of the data, to the development of findings and recommendations. An understandable concern is that if people “study themselves” they will be less objective than if studied by objective “honest brokers.” However, having both external and internal members on a study team alleviates that concern. The internal members serve as a source of validity for the larger group who will implement the findings. The external members bring methodology and objectivity.
4. Record and replicate successful initiatives as opposed to focusing only those that experienced problems
It is human nature to try to understand “what went wrong” but there is an equal amount, if not more, to be learned from studying successes. Performance improvement is more cost effective when it repeats successes rather than fixing missteps. With successes the answer is already known and has been tested, while recommendations about missteps require crafting and testing new solutions, which may or may not work. Crafting and testing is always more costly than replicating.
DIA needs to seek out processes that are working well, and study successful practices to understand the factors that make them work. The findings could then be used in training, stories for in-house publications, and transferred to other units, through peer assists.
The purpose would not be to celebrate these as individual successes or as exceptional events (as in awards) rather to recognize these as processes that could be effectively employed by others.
5. Support Communities of Practice at all levels and provide: 1) software designed for Communities, 2) adequate IT resources to maintain Community websites, and 3) adequate start-up help to build Communities
In most corporations and government organizations, Communities of Practice are the major way lessons learned are moved in real time. For example, the US Army has over 60 Communities modeled on CompanyCommand and sponsored by the Battle Command Knowledge System, at Fort Leavenworth, GE has over 600 Communities, Fluor has over 50 Communities. Within the IC there seems a budding yet limited awareness of the value of peer-to-peer Communities of Practice and certainly little IT support for them.
Unlike communities in other settings where the Core Team can respond to Community needs during off hours, most IC Communities operate on classified networks. Core Team members must manage the Community during regular office hours balancing mission requirements and Community needs. A larger Core Team helps Core Team members better balance Community work with mission requirements.
Provide Core Team members with training in relationship building. One major responsibility of Core Team members is to connect members with each other and to build relationships between the members and the Core Team.
Conclusion
A shift in IC cultural norms to empower employees to take action to implement working level solutions will help to build a more effective DIA. By recognizing the untapped knowledge at the frontline and first supervisory level, peers can resolve issues and craft solutions to the problems they face by creating mechanisms to share knowledge and experience among peers. Organizational leaders can help by encouraging time for such networking and providing resources necessary to enable it.
