Difference between revisions of "IKRIS"

Revision as of 03:49, 8 May 2021

MITRE Report on IKRIS Technology Transfer - John F. Sowa
From: http://www.jfsowa.com/ikl/IKLmitre.doc

According to The MITRE TECHNICAL REPORT  MTR060158,
MITRE Support to IKRIS Final Report by Brant A. Cheikes, Ph.D. from November 2006

Abstract

In November 2004, the Intelligence Community’s Advanced Research and Development Activity (ARDA), which subsequently became the Disruptive Technology Office (DTO), requested that the Northeast Regional Research Center (NRRC) hosted at MITRE provide technical oversight and management of a newly-funded “Challenge Workshop” called  IKRIS:  Interoperable Knowledge Representation for Intelligence Support. 
The IKRIS workshop was chartered to address the following challenge problems: 
(1) how to enable interoperability of knowledge representation (KR) technology developed by multiple organizations in multiple ARDA programs and designed to perform different tasks, and
(2) how to practically represent knowledge that is relevant to intelligence analysis tasks in a form that enhances automated support for analysts. 
This is MITRE’s final report on its activities and accomplishments as NRRC Program Manager for IKRIS.

The major accomplishments of the IKRIS Challenge Workshop are summarized as follows:

• IKL—the IKRIS Knowledge Language. This is the key technical result of the IKRIS Challenge Workshop. IKL is a formally-specified language, based on an emerging ISO standard called Common Logic, into and out of which a variety of distinctly different knowledge representation (KR) formalisms can be translated.
• ICL—the IKRIS Context Logic. ICL is a logic formalism for representing and reasoning about context-dependent knowledge, including alternative hypotheses, points of view, world states and scenarios.
• ISIT—the IKRIS Scenarios Inter-Theory. The Scenarios Inter-Theory specifies an approach to translating among the principal formalisms in current use for declaratively representing processes.
• Evaluation Report. The Evaluation Working Group has produced a report showing that IKL is a sound and effective mechanism for knowledge interchange.
• Capstone Demonstration. The Capstone Demonstration serves both as an evaluation of knowledge interchange using IKL, and as an illustration of the potential Intelligence Community impact of the IKRIS-developed approach to interoperability.
• IKL translators. IKRIS participants at Stanford University implemented a set of software tools for building automated translators into and out of IKL.

• Chapter 1 of this report documents the workshop’s milestones, participants, accomplishments and impact.
• Chapter 2 describes MITRE’s support activities and accomplishments, and
• Chapter 3 presents conclusions, recommendations and lessons learned.

Table of Contents

1 Introduction (pg 1)

In November 2004, the Intelligence Community’s Advanced Research and Development Activity (ARDA), which subsequently became the Disruptive Technology Office (DTO), requested that the Northeast Regional Research Center (NRRC) hosted at MITRE provide technical oversight and management of a newly-funded “Challenge Workshop” called IKRIS:  Interoperable Knowledge Representation for Intelligence Support.  The IKRIS workshop was chartered to address the following challenge problems:  (1) how to enable interoperability of knowledge representation (KR) technology developed by multiple organizations in multiple ARDA programs and designed to perform different tasks, and (2) how to practically represent knowledge that is relevant to intelligence analysis tasks in a form that enhances automated support for analysts.

According to ARDA’s workshop plan, MITRE was to serve as prime contractor for the effort, and was to subcontract with an approved team of scientists and engineers who would produce and deliver the desired technical products.  MITRE nominated Dr. Brant A. Cheikes to serve as the NRRC Program Manager for IKRIS.  Prof. Richard Fikes of Stanford University and Dr. Christopher Welty of IBM Corporation (who together conceived and proposed the original idea for IKRIS) were to serve as the IKRIS Technical Leads (TLs), and would be responsible for guiding the technical efforts of the IKRIS workshop team.

Beginning in December 2004 and continuing through early January 2005, MITRE worked with the TLs and ARDA to define a Statement of Work (SOW) for IKRIS.  ARDA identified two roles for MITRE: (1) to oversee the production of tangible deliverables from the IKRIS program, and (2) to facilitate technology transfer.  MITRE conveyed the revised and coordinated SOW to ARDA on 26 January 2005.  ARDA then released the funds and the IKRIS effort proceeded.  The overarching Project Work Statement (PWS) covering IKRIS was subsequently approved by the Government on 17 February 2005, allowing the official Period of Performance (POP) for IKRIS to run from 14 February 2005 through 1 October 2006 (nearly 20 months).  In mid-September 2006, a no-cost extension to 31 December 2006 was approved, to permit Prof. Fikes, Dr. Welty, and MITRE to prepare IKRIS deliverables and other reports for transfer to the DTO.

This is MITRE’s final report on its activities and accomplishments as NRRC Program Manager for IKRIS.  In this report we will only summarize the technical objectives and accomplishments of the IKRIS workshop team; details of the technical results will be provided in a separate report being prepared by Prof. Fikes and Dr. Welty.  This report will instead document MITRE’s supporting efforts.  In the remainder of this chapter, we document the workshop’s milestones (§1.1), participants (§1.2), accomplishments (§1.3) and impact (§1.4).  Chapter 2 describes MITRE’s support activities and accomplishments, and Chapter 3 presents conclusions, recommendations and lessons learned.

1.1 IKRIS Milestones (pg 1)

The IKRIS Workshop achieved several milestones during its 24-month lifespan:

• Project planning—December 2004 thru April 2005
• Kickoff meeting—25-28 April 2005
• Execution of technical program—May 2005 thru March 2006
• Community meeting—3-6 April 2006
• Capstone Demonstration—December 2005 thru September 2006
• Completion of technical work—30 September 2006
• Production of final products and reports—October 2006 thru December 2006

The IKRIS Challenge Workshop officially began at a face-to-face meeting held 25-28 April 2005 at the Columbia Hilton (Columbia MD). 
In attendance were 34 scientists from industry and academia (most of whom were coming under MITRE subcontract to execute the IKRIS technical program), plus 12 representatives from ARDA and the Intelligence Community. 
At this meeting, the basic organizational structure for IKRIS was established.
It was agreed that IKRIS technical work would be performed by five relatively autonomous Working Groups (WGs), each with a designated WG Leader and with a membership chosen by the TLs based on their assessment of each participant’s unique skills and interests.  

The WGs and WG Leaders were:

• Interoperability: Pat Hayes (Florida Institute for Human and Machine Cognition)
• Contexts: Selene Makarios (Stanford University)
• Scenarios: Jerry Hobbs (University of Southern California, Information Sciences Institute)
• Evaluation: Dave Thurman (Pacific Northwest National Lab)
• Technology Transfer: Paula Cowley (Pacific Northwest National Lab)

During the period May 2005 through March 2006, the five WGs conducted their activities independently, coordinating and collaborating using MITRE-furnished e-mail distribution lists, document-sharing services, and teleconferencing systems.
The IKRIS Executive Team—the NRRC PM for IKRIS and the IKRIS TLs—established a policy of meeting every two weeks (by teleconference) to review technical progress and discuss project finances and other management issues. 
Also on a biweekly schedule, MITRE hosted “All Leads” telecons (bringing together MITRE, the IKRIS TLs, and the Leads of each of the five WGs) to discuss technical activities, schedule, and related issues.
By March 2006, the Interoperability, Contexts, and Scenarios WGs had each completed drafts of their respective technical products. 
A second IKRIS community face-to-face meeting was held 3-6 April 2006 at the Computer History Museum in Mountain View, CA. 
The leader of each WG presented the group’s technical results in plenary, allowing peer review by the entire IKRIS community. 
The ensuing discussions motivated revisions and enhancements to the draft technical specifications, which were then refined over the period May thru September 2006.
In parallel with the core technical development efforts, a “Capstone Demonstration” effort took shape. 
Initiated by the joint efforts of the Evaluation and Technology Transfer WGs, and ultimately directed by Dr. Welty, the Capstone Demonstration sub-project put the fundamental IKRIS-developed interoperability approach to the test. 
Three analyst-support prototype systems—KANI, by the Stanford University, IBM Corporation, and Battelle/PNNL team, Nooscape, by the Cycorp team, and Slate, by the Rensselaer Polytechnic Institute (RPI) team — were selected from the suite of tools that had been developed under the auspices of ARDA’s NIMD (Novel Intelligence from Massive Data) program. 
Common to these three systems was their use of sophisticated knowledge representation and reasoning technologies to assist analysts with various aspects of intelligence reasoning and decision making.

The Capstone Demonstration team developed a realistic intelligence-analysis scenario based on a case study, called “The Sign of the Crescent”, obtained from Prof. Frank Hughes of the Defense Intelligence Agency’s (DIA) Joint Military Intelligence College (JMIC). 
The Capstone team then showed how the IKRIS interoperability solution enabled the three NIMD systems to work together, under guidance from an analyst, to formulate and test several hypotheses that are central to the DIA/JMIC case study.
The Capstone Demonstration activity was originally conceived in late 2005.  It took concrete form in February 2006 when the team was formally established and the broad outlines of the demonstration scenario were defined.  Significant progress was made during the April 2006 IKRIS community meeting, and the Capstone team continued to work with increasing intensity over the summer of 2006, coordinating their efforts with frequent teleconferences. 
A Capstone-specific face-to-face meeting was held on 13 September 2006, at the IBM facility in Hawthorne NY.  During this single focused workday, the Capstone demonstration reached a state of near-completion. 
The team continued to work through September to tie up loose ends and complete the project. 
The Technology Transfer WG delivered a briefing package documenting the demonstration storyboard and illustrating key examples of IKRIS-enabled interoperation among the three systems.
The IKRIS Challenge Workshop completed its technical work by 30 September 2006.  Wrap-up reporting work is expected to continue through December 2006.

1.2 IKRIS Community Membership (pg 3)

Over the IKRIS Workshop’s lifespan, the “IKRIS Community” grew to become larger than just those scientists and engineers directly funded to perform IKRIS research and development.  The complete list (excluding TLs and WG leaders) of all those who participated in the IKRIS technical effort (asterisks indicate those who participated without direct financial support) is below:

Bill Andersen (OntologyWorks, Inc.); *Fotis Barlos (BAE Systems); Danny Bobrow (PARC); Selmer Bringsjord (Rensselaer Polytechnic Institute); John Byrnes (FairIsaac/HNC Software); Alan Chappell (Battelle Memorial Institute/Pacific Northwest National Labs); Andrew Cowell (Battelle Memorial Institute/Pacific Northwest National Labs; Chris Deaton (Cycorp); Keith Goolsbey (Cycorp); Michael Gruninger (University of Toronto); *Ian Harrison (SRI International); Karl Heuer (Stanford University); Robert Hoffman (Institute for Human and Machine Cognition); Mario Inchiosa (NuTech); David Israel (SRI International); Charles Klein (Cycorp); *Hua Li (Sarnoff Labs); Arun Majumdar (VivoMind); David Martin (SRI International); Mark Maybury (MITRE); Drew McDermott (Yale University); Deborah McGuinness (Stanford University); Sheila McIlraith (University of Toronto); Chris Menzel (Texas A&M); Dan Moldovan (Language Computer Corporation); David Morley (SRI International); Leo Obrst (MITRE); Jennifer Ockerman (Johns Hopkins University/Applied Physics Lab); Valeria de Paiva (PARC); Richard Rohwer (FairIsaac/HNC Software); Andrew Shilliday (Rensselaer Polytechnic Institute); John Sowa (VivoMind); Joshua Taylor (Rensselaer Polytechnic Institute); *Marco Valtorta (University of South Carolina); *Russ Vane (General Dynamics Advanced Information Systems); Michael Witbrock (Cycorp); Wlodek Zadrozny (IBM Corporation)

It should be noted that most participants were provided with relatively modest amounts of funding for their efforts, mostly to cover travel expenses, but also to cover some technical labor delivery.  Some of these individuals were able to contribute time significantly in excess of what the IKRIS Workshop was able to fund, and we are grateful for their efforts.  

In addition, the IKRIS Executive team is grateful to our four “Government Champions”, who provided advice and technology-transfer guidance over the course of the project:  Steve Cook, John Donelan, Jean-Michel Pomarede, and John Walker.

1.3 Summary of IKRIS Accomplishments and Impact (pg 4)

These are the major accomplishments of the IKRIS Challenge Workshop:

• IKL—the IKRIS Knowledge Language. This is the key technical result of the IKRIS Challenge Workshop. The Interoperability WG developed a formally-specified language, based on an emerging ISO standard called Common Logic,<super>1</super> into and out of which a variety of distinctly different knowledge representation (KR) formalisms can be translated. Using IKL as an interlingua, knowledge representation and reasoning (KR&R) systems are able to interchange knowledge, inference rules, partial proofs, etc., and thereby carry out fully or partially automated collaborative problem solving. In addition, IKL is a language that knowledge repositories may use for representing reusable knowledge in a way that is understandable by a broad range of KR&R systems.
• ICL—the IKRIS Context Logic. ICL is a logic formalism for representing and reasoning about context-dependent knowledge, including alternative hypotheses, points of view, world states and scenarios. IKRIS produced the ICL formalism, methods for translating knowledge represented in ICL into and out of IKL, and methods for performing effective automated reasoning with context-dependent information represented in ICL.
• ISIT—the IKRIS Scenarios Inter-Theory. The Scenarios Inter-Theory specifies an approach to translating among the principal formalisms in current use for declaratively representing processes (i.e., scenarios). ISIT consists of an ontology of terms for representing processes, and a set of “bridging axioms” for translating to and from other process representations. ISIT is represented in IKL and thereby enhances IKL's capabilities as an interchange language by providing it with a representation vocabulary and translation methodology specifically designed for processes.
• Evaluation Report. The Evaluation WG has produced a report describing a collection of informal evaluations of the core IKRIS interoperability specifications. For example, by performing a series of “round-trip” knowledge-exchange experiments, each increasing in complexity and rigor, the Evaluation WG has shown that IKL is a sound and effective mechanism for knowledge interchange.
• Capstone Demonstration. The Capstone Demonstration serves both as an evaluation of knowledge interchange using IKL, and as an illustration of the potential Intelligence Community impact of the IKRIS-developed approach to KR&R system interoperability.

The Capstone Demonstration team produced
(1) a detailed scenario in which an intelligence analyst develops and tests a sequence of hypotheses,
(2) a storyboard presentation that describes how three distinct KR&R systems interoperate effectively using IKRIS-designed tools and techniques to assist an analyst as she develops and tests her hypotheses, and
(3) a collection of knowledge bases (KBs) serving as a record of the execution of the Capstone storyboard by the three selected interoperating systems.

• IKL translators. IKRIS participants at Stanford University implemented a set of software tools for building automated translators into and out of IKL. They then used those tools to build example translators for use in the Capstone Demo. In addition, the RPI team developed translation software for their Slate system to read and write IKL knowledge bases.

In summary, IKRIS achieved or exceeded all of its planned outcomes.

1.4 Present and Future Impact of IKRIS (pg 6)

The Intelligence Community impacts of the IKRIS-developed technical specifications are significant, though as of this writing, largely awaiting realization.  First and foremost, IKRIS has developed a cross-cutting enabling technology that will facilitate wider use of KR&R technologies across the Intelligence Community.  Heretofore, specialized KR&R system “stovepipes” have been developed, often under IC funding, to demonstrate or deliver specific analysis-support capabilities to limited target user communities.  In many cases, these systems have possessed complementary KR&R capabilities, yet have been unable to leverage each other’s strengths due to their inability to “speak a common language.”  To enable “knowledge interoperability”, these systems need to be able to express their knowledge not only in a common syntax, but also do so in a way that preserves the intended semantics (meaning) of the expressed knowledge across system boundaries.  The IKRIS Challenge Workshop has addressed both aspects of this problem.

The IKL specification appears to fully meet the need for a common syntax, based on the set of alternative KR formalisms considered during the course of the IKRIS workshop.  That is, it is now possible to mechanically translate a well-formed expression in any of the target KR languages studied by the IKRIS workshop team to a well-formed IKL expression, and from there back to a well-formed expression in a target KR language.

The key feature of IKL is its ability to support the transfer of meaning across system boundaries.  As a result, we can now mechanically translate a knowledge base kbA of knowledge structures expressed in the native KR language of KR&R system A into a knowledge base kbIKL of knowledge structures expressed in IKL, and then from kbIKL into a kbB of knowledge structures expressed in the native KR language of system B, such that any sentence logically entailed by kbA also is entailed by kbB.  This means that system B not only can incorporate knowledge transferred to it from system A, but also can perform automated reasoning using that knowledge. 

This ability to transfer meaning across system boundaries is essential because it enables automated collaborative problem-solving by KR&R systems having unique inference capabilities that are intimately tied to features of their internal KR languages.  In essence, IKRIS has set the stage for moving beyond the IC’s current goal of cross-boundary information sharing to the more challenging goal of automated cross-boundary knowledge sharing.
 
Besides developing a specification for IKL that allows transfer of meaning among KR&R systems, the IKRIS  workshop team took IC needs into account when designing IKL.  Three IKL language features represent major technical accomplishments and are worth highlighting here:

1. IKL treats propositions and sentences as first-class objects in the language. This allows KR&R systems not only to express intelligence information as IKL propositions and sentences, but also to represent and reason about meta information, such as the security classification of intelligence information, its provenance, its credibility, its relations to other pieces of intelligence information, etc.

2. IKL supports the expression of relativized names. This feature makes it possible to reason effectively in situations in which one must distinguish between names and their denotations. For example, in an intelligence analysis scenario, we might want to represent and reason about multiple entities (persons, organizations, etc.) which might be known to different people by different names. So we might want to be able to represent the fact that the person whom John believes is called “Mary” is actually the same person whom Bill believes is called “Jenny” —- or that a piece of weapons technology referred to by terrorist J as “the special shipment” is the same thing as “the new product” referred to by arms dealer B.

3. IKL supports the explicit definition of sortal restrictions on existence, and of relationships between sorts or types. This is because IKL content is often reliant upon some framework of classification of things into categories or classes, the primary use of which is to provide appropriate quantifier restrictions. Such a framework of categories is often referred to as a system of types or sorts, and many logics and notations are designed to conform to them, with special mechanisms for handling sortal reasoning or even allowing type checking to be done at parse time. IKL is not a typed logic in this sense, but it allows those restrictions and relationships to be made explicit for purposes of translating content into IKL from such a typed or sorted notation.

Although the value of IKL is substantial, the additional value of both ICL and ISIT should not be overlooked.  The ICL formalism provides a mathematically sound and rigorous foundation for representing and reasoning about alternatives, such as alternative hypotheses, interpretations of facts, chronologies of events, etc.  This technology will allow the development of new analyst-support tools that aid hypothesis generation and testing.  The Scenarios Inter-Theory shows how highly specialized knowledge about time, events, process inputs/outputs and preconditions, and cause and effect relationships can be transferred among systems in a meaning-preserving way.  This is of particular value to the IC given that this kind of “scenario” knowledge is often fundamental to analytic reasoning and decision making.

In the next section, we summarize MITRE’s efforts to facilitate the achievement of these important results.

2 Summary of MITRE Support Activities (pg 8)

MITRE’s activities in support of the IKRIS Challenge Workshop fall into two categories: administration/logistics, and direct technical.  In this section we summarize MITRE’s activities and accomplishments in each category.

2.1 Administration and Logistics Support (pg 8)

As prime contractor, MITRE staff expended significant labor on subcontracting and contracts administration.  MITRE executed Basic Ordering Agreements (BOAs) with twelve (12) corporate entities, and each fiscal year (FY) executed Task Orders against these BOAs.  In addition, MITRE executed direct “consulting engineer” subcontracts with nine (9) researchers.  Over the course of each FY, MITRE solicited and processed invoices, and monitored budget performance.  As fiscal issues or opportunities arose, MITRE developed and executed action plans in coordination with the IKRIS TLs.

In addition to the administration of 21 subcontracts over two fiscal years, MITRE also provided meeting-planning support for all major IKRIS meetings, including the April 2005 kickoff meeting at the Columbia Hilton (Columbia MD), the work session during the June 2005 NIMD Principal Investigators Meeting (McLean VA), the work session during the November 2005 NIMD/ARIVA/Topsail Joint Principal Investigators Meeting (Orlando FL), the April 2006 IKRIS Community Meeting (Mountain View CA), and the Capstone Demo Meeting at IBM (Hawthorne NY) in September 2006.

To facilitate the technical execution of the IKRIS Challenge Workshop, MITRE established an access-controlled, Internet-accessible website for document sharing2,  a collection of archived e-mail distribution lists, a password-protected, Internet-accessible website for browsing the e-mail archives3,  and a public IKRIS website4.   MITRE’s toll-free teleconferencing system was used several times every month to support Executive team, All-Leads team, and regular Working Group technical sessions.  As IKRIS technical materials reached readiness for review outside the IKRIS community, MITRE worked with ARDA/DTO staff to review these materials for public release approval.  MITRE also served as “corresponding secretary” on behalf of IKRIS, responding to inquiries and managing access to the document repository.

MITRE prepared and delivered monthly progress reports to the Government, and beginning in January 2006, met approximately monthly with the cognizant ARDA/DTO Program Manager to provide updates on IKRIS activities and accomplishments.

2.2 Direct Technical Support (pg 9)

While overall direction of the IKRIS technical effort was in the hands of the Technical Leads (Fikes and Welty), MITRE played two supporting roles: 
(1) advising the TLs on technical strategies or decisions which might affect the IC applicability of IKRIS, as well as serving as liaison between DTO and the IKRIS community, and
(2) participating in the Technology Transfer Working Group (TTWG), and leading the TTWG’s IC outreach effort.

MITRE made progress on technology transfer and IC outreach throughout the IKRIS project, with specific activities described below.

• April 2005: IKRIS Kickoff meeting: This was the first meeting of the TTWG. MITRE collaborated with the TTWG Lead (Paula Cowley of Pacific Northwest National Lab) to develop an initial IC outreach strategy. Guidance was obtained from the IKRIS Government Champions (GCs): Steve Cook and John Walker of NSA, and John Donelan and Jean-Michel Pomarede of CIA.
• May 2005: MITRE met with Steve Cook and John Walker at Ft Meade. The purpose of this meeting was to better understand their interest in IKRIS and their ability to facilitate operational use of IKRIS results, and to discuss IC leads for follow-up.
• June 2005:
  • Meeting with John Donelan and Jean-Michel Pomarede at their office complex. As with the prior meeting with Cook and Walker, the purpose of this meeting was to better understand their interest in IKRIS and their ability to facilitate operational use of IKRIS results, and to discuss IC leads for follow-up.
  • Meeting with Lisa Yanguas at Ft Meade. Ms. Yanguas had attended the IKRIS Kickoff meeting in April. We discussed IKRIS objectives and potential IC programs which could benefit from IKRIS results.
  • NIMD PI meeting. MITRE organized the IKRIS evening work session. During the work session, a TTWG strategy meeting was held with the IKRIS GCs.
• July 2005: TTWG telecon. Discussion of TTWG strategy and plans.
• August 2005:
  • Facilitated by Steve Cook, MITRE briefed NSA’s Analysis & Production Technical Directors Committee. This briefing provided early exposure of the IKRIS workshop to a large audience of technical managers.
  • MITRE made initial contact with personnel at the Air Force Rome Lab’s Information Directorate (IF), specifically, IFE—Information & Intelligence Exploitation. This discussion had these outcomes: AFRL staff agreed to join the IKRIS TTWG, and an invitation was extended for IKRIS representatives to visit AFRL and present a briefing.
• September 2005: TTWG teleconference. This telecon included two AFRL representatives (Jeff Hudack and Craig Anken). During this telecon, MITRE discussed initial planning for the Capstone Demonstration.
• October 2005:
  • MITRE developed and disseminated a Technology Transfer WG strategy document that articulated specific WG objectives, identified what was within the WG’s scope, identified measures of effectiveness, outlined requirements for the IKRIS Capstone Demonstration, and identified next steps.
  • TTWG teleconference. During this telecon, the TTWG strategy was discussed and further developed.
  • Capstone Demonstration telecon. MITRE hosted the first planning teleconference for the Capstone Demonstration. The group discussed objectives and the technical approach.
  • MITRE worked with the TLs to plan the first “IKRIS User’s Group” session during the following month’s Joint Principal Investigators Meeting (also called the NIMD/ARIVA/Topsail “Three-Ring Circus”).
• November 2005:
  • IKRIS work session held during the Joint Principal Investigators Meeting, focusing on the plans for and design of the Capstone Demonstration. Also, the first “IKRIS User’s Group” session was held, in which the IKRIS work was exposed to a broader community of DTO contractors and Government representatives. As a result of this session, several new (unfunded) participants joined the IKRIS community and gained access to the draft technical products.
  • MITRE briefed Dr. Art Becker (ITIC) on IKRIS and discussed potential applications of IKRIS results within the Blackbook architecture.
• December 2005: The TTWG and the Evaluation WG worked together to develop and document the first Capstone Demonstration storyboard.
• January 2006:
  • MITRE presented a briefing on IKRIS to an internal Semantic Web community of interest. This identified several MITRE technical staff members who could assist in the technical review of IKRIS products, as well as suggested several leads for follow-up outreach.
  • The TT and Evaluation WGs continued to refine the Capstone Demonstration storyboard.
  • An internal MITRE meeting was held with Dr. Eric Hughes (FFRDC support to ITIC Knowledge Discovery and Dissemination) to discuss tech-transfer approaches into the KDD program.
• February 2006:
  • TTWG teleconference: continued development of the Capstone Demonstration.
  • MITRE presented a briefing on IKRIS at AFRL. This identified some potential applications of IKRIS to emerging AFRL programs, and also yielded a contact to the “Semantic Web in the Intelligence Community” Group (SWIG).
• March 2006:
  • TTWG teleconference: continued development of the Capstone Demonstration.
  • MITRE presented a briefing at the Joint Warfare Analysis Center (JWAC, Dahlgren VA) to the “Semantic Web in the IC” Group (SWIG). This yielded a lead to a DIA metadata program.
• April 2006:
  • Second IKRIS community-wide meeting. During this meeting, a session was held on tech transfer strategy, and a full day was spent on detailed Capstone Demonstration design.
  • Final NIMD Principal Investigators meeting. A second “IKRIS User’s Group” session was held, updating the broader NIMD community on IKRIS technical developments.
  • Steve Cook and John Walker (IKRIS GCs from NSA) facilitated a demonstration and briefing session at Ft Meade covering two NIMD-derived technologies (the KANI and Nooscape prototypes) and IKRIS. This yielded several leads for follow-up outreach.
  • Richard Fikes presented a briefing on IKRIS to an audience at the AAAI 2006 Spring Symposium.
• May 2006:
  • MITRE established an IKRIS document repository on the NIMD Twiki, to support access by the broader NIMD contractor community to draft IKRIS technical specifications. This was in response to a suggestion received during the April IKRIS User’s Group session.
  • Chris Welty met with Lucian Russell at DIA to discuss IKRIS technology transfer opportunities.
  • Work on the Capstone Demonstration resumed by teleconference, picking up where the design work left off in April. Leadership of this effort was taken over by Chris Welty.
• June 2006: Intensive technical development work on the Capstone Demonstration.
• July 2006:
  • Intensive technical development work on the Capstone Demonstration.
  • MITRE presented a briefing on IKRIS to Windy Joy Springs, Technology Development Division Chief, Joint Single Integrated Air Picture (SIAP) Systems Engineering Office (JSSEO), at her office in Crystal City.
• August 2006:
  • Meeting with the CIA GCs to discuss outreach at CIA.
  • Meeting with Jack Lucas of DIA regarding outreach at DIA.
  • Continued development of Capstone Demonstration.
• September 2006:
  • Capstone Demonstration team meeting at IBM (Hawthorne NY). An intensive one-day work session to wrap-up the demonstration effort. A few loose ends were left, to be completed before the end of October.

2.3 Summary (pg 12)

In summary, MITRE contributed to the IKRIS Challenge Workshop by executing and administering over twenty subcontracts, providing remote collaboration services and support to a large distributed research community, organizing meetings, participating in the activities of the Technology Transfer Working Group, and leading or facilitating outreach efforts to the broader Intelligence Community.

3 Conclusions, Recommendations and Lessons Learned (pg 13)

This section presents MITRE’s conclusions from its management and participation in the IKRIS Challenge Workshop, our recommendations to DTO for further activity and investment, and a summary of lessons learned to help improve any future Challenge Workshops that might be planned.

3.1 Conclusions (pg 13)

MITRE expended a significant amount of labor on Intelligence Community outreach. This labor involved online research and personal networking to identify leads to IC programs and associated points of contact, followed by e-mail messages and or telephone calls. A percentage of these leads yielded opportunities to meet and brief, and a percentage of these meeting/briefing opportunities yielded additional leads. Some leads were not pursued, due to limited resources for outreach as well as assessments of likely limited payoff, and other leads had the principal outcome of extending awareness of the IKRIS Challenge Workshop and its expected results.

Based on our many conversations with IC staff during our outreach efforts, we offer these conclusions:

1. The IKRIS project has made a valuable contribution to the long-range goal of creating more powerful and capable analyst-support tools that build upon advanced knowledge representation and reasoning (KR&R) technologies. The significance of the IKRIS technical results is recognized and appreciated within the KR&R branch of the Computer Science/Artificial Intelligence community.

2. The terms “knowledge representation” (KR) and “knowledge base” (KB) have very specific meanings within the IKRIS research community. In IKRIS, KR is a technical discipline in which various kinds of human knowledge are expressed in mathematically precise logical forms which permit the automated generation of new knowledge structures by the application of machine inference techniques; KBs serve as repositories of these precise knowledge structures. In contrast, IC technical staff typically use the term “knowledge base” to mean a conventional (relational) database that serves as a repository of organizational knowledge about intelligence targets; in this parlance, KR is the process of populating an intelligence database. Although exceptions were found, the IKRIS meanings of KR and KB are correctly understood and appreciated largely in niche communities found within IC agency science and technology research directorates, rather than acquisition or operational directorates.

3. At present, there appears to be only one significant operational deployment of an analyst-support system which employs KR&R technologies like those considered by IKRIS. The manager of this particular program has been an active participant in IKRIS, and served as a Government Champion. KR&R technologies are otherwise employed principally in demonstration prototypes developed under advanced R&D initiatives (including DTO programs such as NIMD and AQUAINT). Staff members within several major IC organizations are actively tracking developments in the KR&R arena, with the expectation that as the underlying technologies mature, their organizations will have a clear interest in making use of envisaged applications.

4. IKRIS has focused on enabling interoperability among heterogeneous KR&R systems. This is a significant problem, and the value in solving it is illustrated by the Capstone Demonstration. The IKRIS effort should be understood as having anticipated this problem and developed a solution approach for it well before it becomes an obstacle to the development and deployment of analyst-support tools in the operational IC.

5. Although KR&R is a cross-cutting enabling technology for intelligence analysis support, there is no clear R&D “champion” for it at present in the Intelligence Community. To date, advances in KR&R methods and IC applications have been made in fits and starts under the auspices of research initiatives with broader charters (e.g., the NIMD and AQUAINT programs). As a result, there has been a lack of sustained, IC-focused advancement in KR&R fundamentals, jeopardizing the steady maturation of deployable applications. While the IKRIS Workshop has made an important technical contribution, in the absence of a KR&R champion for IC-focused R&D, there is legitimate concern that the full impact of the IKRIS work might not be realized.

3.2 Recommendations (pg 14)

We offer two recommendations to DTO, with a view towards making progress on the long-range goal of creating more powerful and capable KR&R-based analyst-support tools.

1. Continue IC outreach efforts to raise awareness of IKRIS technical results. Most of the outreach efforts conducted by MITRE were carried out while IKRIS specifications were still undergoing active development, so outreach had the main effect of providing advance notice of accomplishments to come. Now that the work is complete, DTO should make the technical specifications publicly available and accessible on the Internet, on Intelink, and on specific Intelligence Agency intranets. IKRIS technical briefings and User’s Group sessions should be scheduled during future Principal Investigators meetings for DTO programs in Information Exploitation.

2. Convene a strategic R&D planning workshop tasked to develop a research roadmap for further investment in cross-cutting KR&R technologies (including IKRIS-enabled interoperating KR&R systems) that have demonstrable IC applications. Such a workshop should bring together representatives from Science and Technology (S&T), Research and Development, Acquisition, and Operations organizations across the IC, KR&R researchers from academia and industry, and should also include analyst representatives to advise on end-user application concepts. Starting with tutorials on KR&R principles, techniques and existing technologies, this workshop could identify technical gaps and application concepts, and define a sustained investment plan leading to operationally effective KR&R-based analyst-support tools.

3.3 Lessons Learned (pg 15)

Reflecting on our IKRIS experience, we were able to assess a number of organizational and process methods and decisions.  

Some of the lessons learned and/or best practices that may be of interest to future workshops include:

1. Regular Communication. Regular Executive Team and All-Leads meetings facilitated rapid recognition and removal of roadblocks, enabled agile reprioritization and redirection, and in general helped to maintain momentum of technical efforts. Holding these meetings every two weeks seemed to be the right frequency, and they were conducted effectively as teleconferences.

2. Distributed Leadership: By organizing the technical efforts into a set of subgroups with specific charters and respected, capable leaders, we were able to distribute our management responsibilities and push accountability for technical results down to the most qualified experts.

3. Temporally and Geographically Distributed Meetings. The bulk of the IKRIS technical efforts were conducted by participants working remotely from their home institutions, supported by electronic collaboration technology. This allowed us to take advantage of technical talent from across the USA and Canada, while keeping meeting costs to an acceptable level. We changed meeting venues to balance travel demands and costs equitably across the IKRIS participant community.

4. Suite of Remote Collaboration Tools. A suite of remote collaboration tools was needed to effectively support the distributed IKRIS team. E-mail distribution lists were essential; a searchable e-mail archive was requested early on, but it isn’t clear how extensively it was used. Toll-free teleconferencing support was also essential. Web conferencing services were not available until near the end of the workshop, but we suspect that they would have been heavily used if they had been available. MITRE’s document sharing system was also very useful, though it had rather complex mechanisms for user authentication, and reportedly did not work well on non-Windows computing platforms. So future workshops should make early provision for a suite of easy-to-use, platform-independent remote collaboration services.

5. Highly engaged government champions. The government leads were very helpful in focusing, providing opportunities, contributing domain knowledge and experience, and overall cheerleading of the effort.

While every workshop is unique, we hope these lessons will be of use in making future workshops as effective as IKRIS.