Document status: Draft

Abstract

Competitive performance in today’s organization requires a good handle on how to acquire, recognize and use competencies within the organizations. Automated competency tracking and management in the context of performance support, training and adaptive online learning requires a systematic way to define and track competencies for individuals and teams. However, the competency data may come from a variety of sources and in many different formats. This paper proposes a simplified framework that uses simple, standard data formats to help automate the collection and adaptive assessment of individual and group competencies. The framework also supports the automation of skill gap analysis as well as the automation of adaptive performance support. The framework supports not only the automation of various processes, but also well informed decisions by humans. By keeping the data and processes as simple and transparent as possible, humans can apply common sense, priorities and policies with a full understanding of what is going on. At the same time, the framework supports a variety of enterprise policies, as well as credibility issues, sanity measures to avoid corruption by unreliable data, auditing, and general security, confidentiality and privacy requirements. The main goal, however, is to enable this functionality with the simplest possible model while taking advantage of standards that already exist. This white paper may result in specific proposals for new standards or for inclusion in existing standards projects.

Contents

Introduction. 4

Scope. 4

Purpose. 5

How to read this document 5

Acknowledgements. 5

Reality checks. 6

General requirements. 6

Doing more with less. 7

Framework application scenarios. 11

Capturing competency requirements. 11

Defining competencies. 11

Introducing the Reusable Competency Definition (RCD) 14

Introducing the Reusable Competency Map (RCM) 19

Job competency profile. 24

Capturing individual or group competency data. 25

Evidence of competency. 26

Distilling competency evidence. 27

Sanity policy scenarios. 31

Assessment instruments. 33

Data standards from learning technology. 34

Other data standards. 35

Assessment from existing data. 35

Enforcing confidence policies. 36

Complex assessments. 37

Reusing or defining competency definitions. 37

Reporting and analytics. 38

Security, confidentiality and privacy. 39

Personalized assessment plans. 41

Targeted assessments and quizzes. 45

Adaptive assessments. 45

Test item bank automation. 46

Targeted, adaptive learning plans and packages. 47

Targeted learning plans and packages. 48

Adaptive learning plans and packages. 48

Automated and semi-automated learning packages. 49

Dealing with unavoidable changes. 50

Adaptive performance support 52

Adaptive job aids. 52

Just in time training. 52

Data for analytics to support process changes. 53

Putting it all together 54

Appendix A - Data models. 57

Appendix B - Services. 58

Appendix C - Using CORDRA.. 59

Appendix D - References. 60

Figures

Figure 1 - Finding reusable competency definitions or competency models. 11

Figure 2 - Sources of competency requirements. 12

Figure 3 – Capturing competency requirements into standard data formats. 13

Figure 4 – Outline view of a sample competency tree fragment 13

Figure 5 – RCDs are intended to capture only the reusable portion of competency data. 14

Figure 6 – The standard parts of a reusable competency definition. 15

Figure 7 - Reusable competency definitions don't have to be reinvented for each context 16

Figure 8 - Example of advanced search form for RCDs or competency models. 17

Figure 9 - Referencing RCDs in a competency map. 19

Figure 10 – Different competency maps representing different models of the same competency. 20

Figure 11 - Task models can be simple hierarchies of tasks or very complex models. 21

Figure 12 - Capturing existing competency data into the framework. 26

Figure 13 - Summary of the competency evidence distillation process. 27

Figure 14 - Portfolio vs. standard competency records. 27

Figure 15 - Competency record. 28

Figure 16 - Evidence record. 28

Figure 17 – Sources of evidence. 29

Figure 18 – Assessment results. 29

Figure 19 - Enforcing sanity in handling competency evidence from SCORM.. 31

Figure 20 – Distilling competency information from the interview process. 32

Figure 21 – Distilling a single competency record from multiple evidence records. 33

Figure 22 - Table summarizing a sample confidence policy. 36

Figure 23 - Assessment request 37

Figure 24 - From personalized assessment plans to competentency records. 41

Figure 25 - Assessment request data model 43

Figure 26 - Assessment result data model 44

Figure 27 - Combining repositories, competency records and personalized assessments. 45

Figure 28 - Developing learning content 47

Figure 29 - Skill gap analysis. 48

Figure 30 – An integrated competency management, assessment and learning framework. 50

Figure 31 - Performance support 52

Figure 32 – Putting it all together: Simplified framework exploiting standard competency data. 54

Introduction

Scope

This paper proposes a simplified framework that uses simple, standard data formats to help automate the collection and adaptive assessment of individual and group competencies. The framework also supports the automation of skill gap analysis as well as the automation of adaptive performance support. The framework supports not only the automation of various processes, but also well informed decisions by humans.

In scope:

· Basic data models for the representation of competency definitions, simple competency models, and personal competency records.

· Basic data models for some types of assessment data.

· Simple hierarchical competency models.

· Metadata associated with various types of data instances to allow automatic or manual searches and some automated processes.

· Basic automation processes for the collection, distillation and storage of competency data for individuals and groups.

· Basic automation processes that exploit competency data, such as competency profiles, readiness reports, skill gap analysis, personalized assessments, and personalized performance support.

· Personalized, adaptive training based on performance requirements and personal profiles.

Out of scope but assumed to exist

· Processes by which real world requirements expressed as operational objectives, policy directives, business process documents and technical documentations are massaged into performance specifications and competency definitions.

· The actual construction and maintenance of task models, ontologies, complex competency models. The framework assumes that those can be stored and exploited, but does not specify how to construct or maintained them.

· Specific storage technology, repository configuration, or network topologies.

· Personal portfolios and other personal profiles, such as would be maintained by a human resources department or in school registration records.

· Specific security infrastructure, processes and policies. The framework is however designed to facilitate the application of security, confidentiality and data sanity policies.

Out of scope and not required by the framework

· Artificial intelligence and related data collection and processes.

· Massive, all-encompassing and immutable competency models

Purpose

In organizations and enterprises that require many people to be ready to perform a variety of more or less specialized tasks in order to fulfill the objectives of the enterprise, the quality of training, as well as the costs and time sunk into training, have major impact on performance. Competency assessments, whether it is to find right people or teams for the job or to determine training needs, is difficult without relevant and usable data. Effective training and performance for large numbers of people is a huge problem that can, at least in theory, benefit greatly from automation, if only by ensuring that the correct training is targeted and that no time is wasted on unnecessary training.

Although the requirements in education or in executive development programs have different names, in effect they are very similar. Achieving verifiable learning outcomes, domain-specific competencies, and general competency as a well rounded human being are among the top stated goals of educational institutions and human development programs. This must be done with ever shrinking budgets in the face of mounting expenses. At the same time, it is more necessary than ever to be able to demonstrate performance in the learning enterprise, in order to secure ongoing funding. Also, the outcomes to achieve must be expressed in a form that requires all participants to understand what is going on so that they can keep the humanity at the core of increasingly automated processes.

The purpose of the proposed framework is to provide the benefits of automation in support of these requirements, without falling into the trap of excessive complexity or requiring wholesale and immediate conversion of existing data and processes. The framework must also provide for meaningful human oversight over the processes, and the application of policies that may vary greatly among communities of practice.

How to read this document

Although the complete framework may appear to be complex, because of the number of different processes it supports, in practice its parts and processes are simple, and no more complex than what people are faced with in normal business, training or education today. In some cases they are simpler than some current models, which tend to be accretions of features and mismatched data.

The data and processes defined in the framework are designed to remain understandable to normal human beings. Therefore, it is recommended that the readers of this document focus on specific scenarios, one at a time, rather than trying to understand the whole framework immediately.

Acknowledgements

This paper draws on the wisdom, ideas, examples, insights and challenges from many people over the course of several years. Without them this proposed framework would not exist. Some people may not have realized how helpful even a brief remark or comment was. Others contributed major work that was seminal in various aspects of this project. A proper bibliography would far exceed in length the paper itself, but would still not capture how conceptual connections were born from the clash and intercourse of ideas in many, many books, articles, conversations, exchanges, demonstrations and encounters with a wide variety of people in schools, meetings, conventions, offices, airplanes, trains, laboratories, work sites and factory floors. Many names come to mind, and I must beg forgiveness for not remembering all of them, so I’d better stay out of trouble by citing none. To all of you, I am humbly grateful.

Reality checks

This paper is grounded in reality. It is also much easier to understand through examples, in the context of some real world scenario. Wherever possible, concrete examples illustrate concepts. The framework is designed to deal with very large automation projects as well as small ones; many of the examples in this paper will be based on the following scenario:

Corporate policy, safety regulations and insurance coverage conditions require that employees know what to do in case of an emergency at the office. Such emergencies include natural disasters, fire, sprinkler malfunction, injuries, health emergencies, workplace violence and terrorist actions.

It is therefore a competency requirement that all employees be ready to behave in any workplace emergency situation according to company policies and procedures.

General requirements

Need for performance

Most enterprises have some common basic drivers. One of them is a need for performance—the ability of the employees or staff to achieve the business objectives of the enterprise. Another one is cost containment. The business objectives must be achieved in the most economical way possible. A third is motivation. Most managers recognize that a well informed, well motivated, confident work force is more likely to produce good results.

What then is performance? It not just the achievement of an objective, it is the ability to do so without excessive cost or effort. One useful definition of performance is as the successful application of the competence of individuals and teams in the completion of a task or the pursuit of an objective.

Problems

To date much of the automation in training and performance has been fairly ineffective, or too expensive to achieve. Typical problems include:

· Poor, incomplete or belated alignment of the training with evolving business requirements.

· Overly complex and rigid systems that become so complicated that no one understands all their parts, and that any improvement is inordinately expensive.

· Inability to capture existing processes and legacy data.

· Disappointing results from promising technologies, such as artificial intelligence. In reality, the knowledge capture process required to support artificial intelligence has turned out to be much harder than anyone expected, and this often limits the scale of projects.

· Dependency on “perfect” data with full referential integrity, when in reality the available data is often less than perfect or may be mismatched, and processing accidents do happen.

· Lack of support for auditing and robust security

· Lack of support for recovery from process errors, bad data or human cheating.

· Ever shrinking funding in the face of ever increasing demands for performance and ever more complex competencies to support performance.

· Secrecy and “Not Invented Here” attitudes leading to constrained functional and data silos that resist integration with other systems.

· Lack of interoperability standards, or lack of awareness of the standards, leading to more or less absolute dependency on particular vendors or on proprietary implementations.

Doing more with less

This paper proposes a framework to support automation. Automation supports decisions by humans. By keeping the data and processes as simple and transparent as possible, humans can apply common sense, priorities and policies with a full understanding of what is going on. At the same time, the framework supports auditing and the automated application of policies to help ensure consistency in the data and in the way the framework is applied.

Doing more with less complicated models and implementations is a driving design principle of this framework. Of course, actual implementations may add value by providing better tools, better interfaces, and especially by providing content in the form of competency definitions, competency models, learning content, and so on.

For each of the problems cited above, the framework proposes a solution or at least, within the constraints of feasibility, an approach to a solution.

Problem	Framework solution
· Poor, incomplete or belated alignment of the training with evolving business requirements.	Capture definitions of the enabling competencies required to achieve performance outcomes. Automation of adaptive training based on skill gaps that may be evaluated in real time according to changing needs. Automation of the specification of required training or performance support materials. Automatic matching of training resources to the competencies to be developed. Support for training resources that can evolve from “good enough” to “great”, depending on time and resources available. Support for different types of learning resources or for choices of training methods. Support for human oversight and override of any automated part of the process.
· Overly complex and rigid systems that become so complicated that no one understands all their parts, and that any improvement is inordinately expensive.	Support a service oriented architecture (SOA) built around business processes and services rather than around a monolithic core. Simple data models that can be combined in powerful ways. Simple “pipeline” and algorithmic processes that can be combined in powerful ways. Small number of predefined service interfaces to reduce the number and complexity of security and privacy filters. Clear definition of processes subject to the application of customized policies, e.g. what kind of competency evidence to trust. Straightforward data stores that can be implemented using generic industry standard technologies Data stores through catalogs that can be presented in a form familiar to any user of commercial web sites like Amazon or eBay, and searched easily using familiar search interfaces.
· Inability to capture existing processes and legacy data.	Provisions for a “distillation” process to distill legacy data into standard data that supports the automation features of the framework. Provisions to retain audit trails and access to the rich data that are not preserved in the distillation results. Resilient design that can function with incomplete data or with data of variable trustworthiness.
· Disappointing results from promising technologies, such as artificial intelligence. In reality, the knowledge capture process required to support artificial intelligence has turned out to be much harder than anyone expected, and this often limits the scale of projects.	No dependency on artificial intelligence. Note that artificial intelligence can be used to mine data and extend the framework in various useful operational ways.
· Dependency on “perfect” data with full referential integrity, when in reality the available data is often less than perfect or may be mismatched, and processing accidents do happen.	Resilient design that can function with incomplete data or with data of variable trustworthiness. “Best effort” design that anticipates data quality problems and simplifies data models to allow easy human detection and auditable correction of errors.
· Lack of support for auditing and robust security	Support for full audit trails for any data created or modified in the framework. Preservation of audit trails in automated processes. Allow auditing of arbitrary decisions, such as competency equivalency decisions. Small number of service interfaces to reduce the number and complexity of security and privacy filters. Clear definition of processes subject to the application of customized policies, e.g. what kind of competency evidence to trust. Supports closed-circuit implementations (e.g. offline, closed intranet) as well as highly distributed implementations. Implementations may be federated trough narrowly defined, easy to secure “pipes”.
· Lack of support for recovery from process errors, bad data or human cheating.	Allow automated after the fact corrections of data records that may be influenced by bad data introduced in the system. Provides for after the fact correction of results influenced by cheating.
· Ever shrinking funding in the face of ever increasing demands for performance and ever more complex competencies to support performance.	Highly scalable using the same models; no need to overbuild to get benefits from the framework. Data and processes can be migrated to implementations of any size without data loss. Simple data models that can be combined in powerful ways. Simple “pipeline” and algorithmic processes that can be combined in powerful ways. Small number of service interfaces to reduce the number and complexity of security and privacy filters. Clear definition of processes subject to the application of customized policies, e.g. what kind of competency evidence to trust. Straightforward data stores that can be implemented using generic industry standard technologies.
· Secrecy and “Not Invented Here” attitudes leading to constrained functional and data silos that resist integration with other systems.	Support for a variety of policies specific to a community of practice. The policy implementations can remain highly customized to support special requirements. Reduce design costs for generic functionality.
· Lack of interoperability standards, or lack of awareness of the standards, leading to more or less absolute dependency on particular vendors or on proprietary implementations.	The framework is based on existing, emerging or proposed standards. The design is largely independent of any implementation or extension. The standard features of the core data models and processes remain functional without proprietary extensions. The framework does not depend on a specific vendor or technology platform (e.g. Microsoft, Sun or *nix).

Framework application scenarios

To facilitate understanding, the framework will be described in the context of major application scenarios. These major applications are:

· Capturing competency requirements(page 11)

· Capturing individual or group competency data (page 25)

· Targeted and adaptive assessments (page 41)

· Targeted, adaptive training and learning (page 46)

· Competency-driven adaptive performance support (page 49)

Capturing competency requirements

Competencies in the example competency scenario

The introduction stated that many examples would be based on the following scenario:

Corporate policy, safety regulations and insurance coverage require that employees know what to do in case of an emergency at the office. Such emergencies include natural disasters, fire, sprinkler malfunction, injuries, health emergencies, workplace violence and terrorist actions.

It is therefore a competency requirement that all employees be ready to behave in any workplace emergency situation according to company policies and procedures.

Firs, we will look at how the framework allows us to capture the detailed competency requirements to support the scenario.

Defining competencies

Chris, compliance manager, asks Don to put together a plan to bring all employees and new hires to a reasonable level of competency in handling emergencies at the office. After some investigation, Don realizes that no one in the organization really knows what this really means.

Don turns to a competency management system. This competency management system provides access to repositories of published of reusable competency definitions and competency models. Using a search similar to Google’s advanced search, he uses a simple form to look for existing competency definitions, using the keywords “office emergency”. He also checks the option to find competency definitions for which a competency model with more detail exists.

This first search returns a list of competency definitions published by various entities. Some of them are free; some of them require a payment to see in detail. All of them have an associated competency model, but the prices for the competency models range from free to hundreds of thousands of dollars. Don chooses to order his search results by price of the competency model, and chooses one that looks like it was published by a reputable source.

Figure 1 - Finding reusable competency definitions or competency models

Competencies are a very vast subject complicated by very strong opinions and cultural traditions. High level competencies are highly personal and essentially impossible to define and describe—the sum is greater than the parts. For example, the competency of a charismatic leader can be analyzed, but such an analysis will never capture some intangible aspects of what makes this person capable of such performance. However there is a lot about competency information and competency data that can be described or captured to enable systematic performance improvement, performance support and training. Standard data models and clearly defined processes or workflows can then enable various forms of automation, such as computer storage of personal competency profiles, skill gap analysis, individualized performance support and training, and decision support for individuals and their managers. For example, such data can be used to find learning resources designed to build the specific competencies required for a task.

The real world as source of competency requirements

Figure 2 - Sources of competency requirements

Normally, competency requirements are drawn from more or less explicit real world requirements. The processes used to analyze those requirements vary from simple head-scratching and pure guesswork to very elaborate, highly validated workflows that result in complex semantic modeling of the problem space. This framework does not attempt to define what such a process it, or what the process involves. However, this framework assumes that it is possible to distill the results of the process into data that can be captured using standards-based data formats and models, and that distillation can be an ongoing process as things evolve.

The distilled data used in this framework may not be as complete as more elaborate descriptions of the real world, but they should be sufficient to support a number of useful scenarios at a reasonable cost. They should also make it easy to deal with inevitable changes over time.

Figure 3 – Capturing competency requirements into standard data formats

For example, analyzing the competencies required for the example scenario might first result in a hierarchy of component competencies, as shown in Figure 4.

Can handle workplace emergency

Knows and understand general emergency policies

Knows what constitutes an emergency

Knows who is responsible in an emergency

Can evacuate safely and follow proper procedures

Knows the evacuation routes

Evacuation procedures

Observe proper priorities in emergency situation

Can cite the order of priorities

Demonstrates understanding that personal safety comes first in emergencies

Demonstrates attention to ensure safety of other people in emergencies

Knows what to do to protect confidential information in emergencies

Can apply emergency property stop loss measures

Can handle a fire event

Demonstrates understanding of fire situations

Knows what to do if you detect or find a fire

Knows what to do if there is a fire elsewhere in the building

Can evacuate safely and follow proper procedures

Knows what to do immediately after a fire that caused damage

Can handle a survivable earthquake event

Demonstrates understanding of earthquake situations

Knows what to do during an earthquake

Knows what to do immediately after an earthquake

Can evacuate safely and follow proper procedures

Knows what to do immediately after an earthquake that caused damage

etc., etc., etc.

Figure 4 – Outline view of a sample competency tree fragment

Figure 4 is a fragment of a competency model in the shape of a tree or hierarchy. There are other ways to model competencies, such as a semantic network or ontology. However, when you take a small subset of a domain a tree like this one is often sufficient. Higher level competencies can be decomposed into competency facets—knowledge, behavior, skill—and more specific sub-competencies.

The tree can of course be extended into more detail, down to specific skill and knowledge definitions, such as “recovering and packaging paper documents damaged by water”.

Note that this tree is not a true taxonomy, because the same competency appears in more than one place in the tree. In this example, knowing the evacuation routes and the proper evacuation behavior are competencies that appear again and again in the collection of competencies required to handle different kinds of emergencies.

Each of the competencies listed in the tree can be defined in more detail. This can take the form of some extended description, statements of knowledge such as “can cite the name and phone extension of the current floor warden”, or some statement of performance like “given a fire somewhere else in the building, follow proper procedure for immediate safe evacuation and regrouping at a predefined meeting point without adding to danger to self or others.”

As we saw above, the same competencies may appear in more than one place in the competency tree. Thus, it makes sense to capture the definitions of those competencies in some reusable form, so they have to be defined only once.

Introducing the Reusable Competency Definition (RCD)

A reusable competency definition (RCD) captures the part of competency information that may be reused for more than one person in one or more contexts and possibly with different metrics.

Figure 5 – RCDs are intended to capture only the reusable portion of competency data

A reusable competency definition (RCD) may define a skill, knowledge, aptitude, or a learning objective. The definition may be for a facet of competency (e.g. affective, psycho-motor, cognitive facets) or for a component competency of a larger competency (e.g. “Knowing the evacuation routes” is part of the larger competency “Able to handle office emergencies”).

The making of a Reusable Competency Definition

Each RCD has at least two parts: An identifier that makes it globally unique and the "payload" that contains the actual definition information. The identifier is intended for machines. The payload is intended for humans. The payload includes data like title, description, formal definition(s) according to one or more community-specific models, and intrinsic metadata about the definition itself, such as when it was created, and so on.

Figure 6 – The standard parts of a reusable competency definition

A RCD must have at least an identifier and a title. The globally unique identifier of a RCD works like the globally unique ISBN number of a book. For most automation scenarios, you only need the identifier. Using an identifier rather than the title or content of the RCD removes any ambiguity when you refer to a particular competency definition in automatic operations. Book lists typically include ISBN numbers, one purpose of which is to avoid confusion between similar sounding titles or different editions of the same work. In the same vein, an inventory of competencies in a personal profile might use the RCD identifiers, so that there is no confusion between that might arise from similar titles.

As for a book, once a RCD is published it should not change. If a change is necessary, this is in effect a new version and the new version requires a new identifier, just like publishing a new version of a book requires a new ISBN. As with books, you may run across many copies of what looks like the same RCD; as long as the identifier is the same you know that it is exactly the same version of the same RCD.

Of course, in practice it will often be necessary to know whether a RCD has been replaced with another one. The proposed framework includes ways to do this without modifying the old RCD.

Metadata provide a means to identify the source of the definition. Metadata can also be used to capture known fixed relationship with other definitions or contexts. For example, if a new RCD replaces an obsolete RCD, the old RCD may be referenced in the metadata. This information can then be used by RCD repositories to automatically add references the new RCDs in the catalog entries they use to describe the older RCD records. There are many other use scenarios for the metadata that can be described elsewhere.

It is important to distinguish between intrinsic or embedded metadata, and external metadata. For example, if you open a book to the copyright page, you will see embedded metadata. The metadata cannot be changed without destroying the integrity of the book. But the metadata can be extracted and put in external metadata records, for example in a catalog. Unlike the embedded metadata, the external metadata records may contain additional, changeable metadata. For example, external metadata may include user reviews and references to new editions of the book. There may be different external metadata records describing the same RCD, just like there may be different external metadata records describing the same learning object. The embedded metadata never changes. The external metadata may change depending on who is describing the RCD and for what purpose.

Since catalogs are collections of external metadata, when a RCD becomes obsolete, repository catalogs can point to the new RCD without modifying the existing RCD or its embedded metadata. The addition of the reference to the new RCD only affects the external metadata captured in the catalogs. If RCD identifiers are handles, as defined in the Handle system (see http://handle.net ) or the handle-based CORDRA specification (see http://cordra.net), the handle itself can be used to find a reference to the current version. How to do this with CORDRA is not entirely defined yet, but is likely to be defined in the months to come.

New competency definitions may be created directly using the RCD standard data model. Many existing competency definitions that exist in various competency models can also be captured into that standard data format. In some cases the existing competency definition may have to broken into several RCDs . The RCD in itself is not sufficient, nor is it intended, to capture the complexity of a competency model. Rather, it is intended as a basic building block to define a competency or facet of competency at any level of granularity in a competency model.

The RCDs data model is defined by existing standards. There is an IMS specification, on track to become IEEE standard, which is under consideration for a possible ISO standard.

For more information about the standards, see http://www.ieeeltsc.org/wg20Comp/ or IMS RDCEO specification at http://www.imsglobal.org.

Reusable Competency Definitions in different contexts

As Don explores the competency model he found in the repository, it becomes obvious that some of the competencies defined in the model won’t apply to his organization. However, some are generic enough to apply just about anywhere.

Figure 7 - Reusable competency definitions don't have to be reinvented for each context

The term "reusable" in “reusable competency definition” does not imply that every RCD is reusable. Rather, it implies that it allows competency definitions captures using the RCD data model to be reusable, if it makes sense. The same data model is used, regardless of the scope of reusability. For example, a particular skill definition may be extremely specific to a particular piece of equipment, with very specific performance criteria. But it can be captured and referenced using the same data model as a more generic definition, such as "able to resolve conflicts rather than escalate them". Using the same data model allows the automatic processes and the data management to be simpler.

Reusable Competency Definitions and quality

The RCD standard is a technical standard and does not specify the quality of the content of a RCD. Similarly, ISBN is a technical standard for book identifiers, and it is entirely silent about the content or subjects of the book. The usefulness of this kind of value neutral technical standard should be obvious. For example, the technical standards for email allow all kinds of messages to be created, transmitted, stored and referenced, regardless of whether the messages are business memos, love letters or spam, short notes or lengthy detailed essays.

Of course, communities of practice that create and use RCDs will have expectations of quality and will probably set their own editorial quality policies. However, one should probably not be to restrictive. After all, there is a lot of value in citing other works in documents, even if those works don’t follow the same editorial guidelines or philosophy. We all buy other people’s products even if they are not exactly what we would build ourselves if we had the time and resources to do it. Reuse sometimes involves compromises where the benefits of reusability outweigh the desire for perfection.

Reusable competency definitions as valuable intellectual property

Because it takes time and effort to create useful RCDs, this creates value. There is no reason why published RCDs might not be copyrighted, published and licensed like other intellectual property.

Finding reusable competency definitions

In order to reuse RCDs, you muse be able to find them. RCDs can be stored in repositories that, like any repository for digital object, may be standalone or federated. Typically, repository functionality includes a way to search the content. This could take the form of a catalog that contains metadata about the actual content. For example, the World Wide Web is a huge repository which can be searched by searching the Google index. When you do a Google search, you don’t search the actual web pages. Instead, you search the Google index, which is a catalog of many pages on the Web. The search results shown by Google include a fragment of the catalog data to help you decide whether what was found is what you want.

Finding RCDs can occur in many ways, such as, for example:

· Searching or browsing by title, or by title and description

· Searching by specific aspects of the definition, such as a specific statement of behavior that exists when a particular model is used in the formal definition part of the RCD,

· Searching for specific characteristics described by embedded or external metadata, such as a particular publisher or a link to related RCDs.

· Finding a RCD automatically through a reference in a competency map or model.

Figure 8 - Example of advanced search form for RCDs or competency models

Trade secrets and classified competencies

As he identifies the competencies required for the handling of office emergencies in his organization, Don stumbles upon a procedure manual with a fat label on the front that says “Confidential information – Authorized eyes only”. The manual describes procedures to shut down and restart the security system in case of emergency. Anyone reading this manual would become aware of the security measures in place in the organization, and of the vulnerabilities that might arise in an emergency. Don needs to include this in his competency model, but at the same time prevent unauthorized people from viewing the actual definitions. The competency management repository he’s using lets him do that. He logs off and logs back in as a guest user, and when he tries to browse the competency definitions that are invisible to unauthorized eyes all he finds is meaningless, opaque identifiers.

The actual competency definitions may be a trade secret, a national security secret, or may have to be kept confidential for some reason.

Many automated operations are enabled by using a standard identifier for RCDs. For example, you might make a list of competencies that are required for a mission. That list could just be a list of opaque identifiers. Unless you are authorized to look up the full RCD records in a repository where the actual RCD content resides, you would have no way to know what the competencies are about.

A personal portfolio might include competency records, which consist of a collection of RCD identifiers with associated proficiency and evidence data. Those records in the portfolios of many individuals can be matched automatically against the list of competencies required for the mission, to generate a short list of qualified individuals, without ever knowing what the competencies actually are. Similarly, a learning management system might keep track of skill gaps and learning assignments without knowing what the assignments are about, because it just juggles records containing opaque identifiers. An authorized manager or learner, however, would typically be allowed to see the actual data that can be retrieved using the opaque identifiers. For an authorized user, everything is visible. An unauthorized user would only get meaningless identifiers.

Introducing the Reusable Competency Map (RCM)

Each of the competencies or competency component in the competency model outlined in Figure 4 on page 12 can be defined in a RCD. We saw that the same competency appears more than once in the model. In our example, “knowing the evacuation routes” appears more than once in the competency model outline. To avoid unnecessary duplication, instead of being embedded directly in the hierarchical structure, the RCDs can be in a separate collection.

If the nodes in the hierarchy just reference the RCDs, but do not actually contain them, we can say that the hierarch is a map of the RCDs. Figure 9 shows how such a structure can be represented. The RCDs are shown round as target symbols, and the hierarchy of competencies is shown as a tree-like structure. The collection of RCDs might not have any particular ordering or organization, but the competency map specifies a particular way to organize a collection of RCDs into a coherent structure of related definitions.

Figure 9 - Referencing RCDs in a competency map

Each node in the model may reference a specific RCD, but some nodes might exist only as a way to group related competencies under a common title. We will see later why this can be useful, for example when gathering assessment results or making up the title for a training course.

Reusable Competency Maps

A Reusable Competency Map is a competency map that represents a particular way to define a structured competency model, or part of a model, that uses a collection of RCDs. For example, “Can evacuate safely and follow proper procedures” includes several sub-competencies and facets. The relationships between those can be specified by a competency map in the shape of a tree where the root maps to the RCD for “Can evacuate safely and follow proper procedures”. Since this competency appears several times in the office emergency handling competency model, this smaller map can then be used as a reusable building block.

Like a RCD, a reusable competency map can be published, have metadata, and so on. There is currently no published standard for such competency maps, but there is a solid proposal which underlies much of the functionality of this framework.

Defining simple competency maps

Don shows the outline of the competency model he found to Chris. Chris scans it and points out that the model does not seem to include elevators, probably because it was designed for a company in one story tall building. Also, the model puts too much emphasis on earthquakes, which are much lower on the risk ranking in the organization’s internal policy statements.

Chris asks Don to fix this before proceeding. Don decides that the easiest way to proceed is to cannibalize what he can from the generic competency model, and to create a competency model that is appropriate for his organization. Using a drag and drop interface, he quickly begins his new model by dragging the competencies he wants to keep from the existing model into the outline of his new model. He adds a node to his tree for elevator in emergency and another one for elevator safety, and prunes some earthquake topics. He searches the repository catalog for competency definitions using the keywords “elevator and emergency” and finds a useful definition for a relevant competency, along with an associated small model that includes knowledge and behavior facets of why and how to avoid being trapped in an elevator in an emergency. He drags that model into his competency model, replacing the placeholder for “elevator in emergency”. He then does a search for elevator safety, finds out that it really does not say much that is relevant to handling emergencies, and decides to delete the placeholder for elevator safety from his competency model.

A competency map is a map of RCDs. It is not made of RCDs, but rather it shows how RCDs are related. It is a structure collection of nodes that reference RCDs.

· The simplified competency map model proposed in this framework is a hierarchy of nodes that refer RCDs. The child nodes of any node can represent facets or component competencies, or both. Such maps are called “taxonomies” in some communities of practice in the fields of competency or recruitment, even though they are not necessarily true taxonomies.

· The same RCD may be referenced by more than one competency map, and/or by more than one node in a competency map (see Figure 10).

· Different communities of practice may structure hierarchies of components and/or facets of a competency in different ways. This is a reality that cannot be overcome. Embrace it, don't fight it.

· Competency map provides part of the context for a RCD. For example, it can help assess a competency in the context of related competencies.

· One or more competency maps may be specifically referenced in metadata for a RCD, using a classification element. This may even specify a "taxonpath" that specifies the path to the node that references this RCD in the taxonomy.

Figure 10 – Different competency maps representing different models of the same competency

Ontologies vs. taxonomies

Competency maps can also be represented as directed graphs. They can be semantic networks that show the often intricate relationship between competencies and their facets and component competencies. However, while very small ontologies look easy, considerable effort is required to scale up to larger models. Useful ontologies are hard to build and even harder to keep up to date. Some interesting work is in progress on building ontologies as the organization mechanism for RCDs, but it may be a while before this work comes to fruition.

Until then, practically speaking, if an ontological or semantic approach is desired, it is probably more useful to build task models as ontologies, and then use that as the blueprint from which to identify relevant competencies or competency clusters that can be described by RCDs and simple tree shaped competency maps, than to start from a theoretical ontology model of the competencies. In practice, then, competency maps should be derived from the task models rather than the other way around.

Task models

Task models come in many different flavors, according to many different communities of practice. Some are hierarchical, others are ontologies. Some define a qualification; some are specific to a domain, e.g. "Radio equipment operation". Some are very specific, e.g. "repair automobile automatic transmission model XYZ", others can be very vast and describe an occupation, e.g. "Accountant" Some are defined as workflows; others, like the US Army MOS, are defined as hierarchical layers of disparate elements.

Figure 11 - Task models can be simple hierarchies of tasks or very complex models

It does not appear possible to standardize task models across communities of practice. However, the existence of more or less explicit task models is assumed in this paper. In the sample scenarios that were examined to validate the concepts in this paper, consideration was always given to how standard data and processes can be applied to real world task or workflows.

In the sample scenario assumed by this paper, a task model would probably start from a description of a particular kind of emergency unfolds. From that, one or more task models can be built. There might be slightly different task models for the floor wardens and for normal employees, for example. By analyzing the task model, it is possible to define the competencies required to accomplish each of the tasks.

Hierarchical competency maps are best for limited domain scopes

At this point, Don decides he needs to have others review his work for sanity. He selects the heading “Observe proper priorities in emergency situation” in his competency model and chooses a menu option to generate a detailed document version of it, including the competency definitions for each level of detail, and sends it to the head of HR asking for a review. He repeats this with different headings, and sends the generated documents to other subject matter experts.

Don gets feedback from HR. There is concern that the way understanding priorities is defined in the model may confuse people, because a statement the competency definition references a policy document that does not exist in this organization. Don decides to create a new reusable competency definition and point the” understanding priorities” heading to that new reusable competency definition instead of the irrelevant one.

Reality is very complex, and in many ways a hierarchical model of competencies is an oversimplification. Hierarchical maps are very useful, because of their simplicity. But they are probably best used to represent limited competency domains, or parts of domains. Experience has shown that massive taxonomies are extremely hard to build and maintain. Perfect large competency taxonomies will probably never exist.

On the other hand, small hierarchical competency maps are easy. So keep it simple and easy. Rather than one big hierarchy, use a lot of smaller hierarchies that are easy to build, easy to review for relevance and accuracy, easy to update. The smaller competency maps can be combined in useful ad-hoc ways when needed.

There is at this point no standard defining how to build competency maps; however, there are several potential candidates, of which the simple hierarchical competency map described in this framework is one.

Why reuse competency maps

Like RCDs, useful competency maps may require some serious work, and that makes them valuable. Therefore it makes sense to consider some competency maps as reusable, and provide a standard way to capture them into standard data objects that can be stored in repositories, published and licensed.

There is another aspect of reuse, even if a competency map is just ad-hoc and not intended to be reused in different contexts. An ad-hoc competency map may be used to represent a structure for the competencies to be tested in an assessment, and this structure can then be used to guide how the results from the assessments of individual competencies can be weighted “rolled up” into a broader assessment result. To support such uses without duplicating the competency map everywhere, a competency map must be a data object that can be referenced and stored in a repository for as long as any other data depend on it.

Reusable competency map metadata

Like RCDs, reusable competency maps can include intrinsic embedded metadata and external metadata. This makes finding, storing and publishing RCMs as simple as finding RCDs or learning resources. Thanks to the metadata, a RCM can just be another kind of thing you can look for in a catalog.

Reusable competency maps for normal human beings

Tree-shaped reusable competency maps can be represented in a computer interface and in reports in the familiar form of an outline as in Figure 4. However, a computer interface would typically allow the user to “inspect” any heading of the outline to look up the corresponding RCD. Behind the scenes, this means that when the user inspects the heading, the identifier of a RCD that is attached invisibly to that heading is used to look up the actual RCD and display it to the user. A typical user interface to explore a competency model might look like the familiar two pane view, with a tree in the left pane and a display or work area in the right pane.

If a competency map is structured as a directed graph rather than a tree, to embody an ontology or semantic mapping, the representation for normal human beings is much more complicated. Whereas navigating a tree is now a familiar metaphor for most computer users, navigating a directed graph along a variety of relation links is still a daunting problem. Most demonstrations seen so far require a grasp of the conceptual foundations of semantic mapping and mental models that are still foreign to normal people. There is a lot of research going on in various places to find a solution to this problem. Until this problem is solved, it is recommended that implementations of the framework rely on tree-shaped maps only, while allowing experimentation with directed graphs.

Adapting for changes in business performance requirements

A new directive is promulgated in the organization. Every office emergency must be reported and must be followed up with an evaluation of how it was handled.

Don realizes that he needs to extend the competency model for handling office emergencies. He adds a couple of placeholders in his outline of the model, while he ponders his next move.

Flexibility is no longer a luxury. It is a basic requirement. Business requirements change, tasks change, equipment changes. These changes must be reflected in competency models used for assessment and training. The simpler the model, the easier it is to make the changes. The framework facilitates changes in several ways:

· Capturing competency requirements in RCDs and competency models

· Decoupling the RCDs and the competency map that represents the competency model structure

· Using small competency models that can be recombined, rather than some monolithic, massive competency model.

By decoupling the RCDs and the competency model structure, we are adding a lot of flexibility. For instance, the example competency model to handle office emergencies can easily be cannibalized to adapt to a different office location where the escape routes and some other local conditions are different, by pointing to different RCDs that are specific to different locales.

Using small competency models that can be recombined helps with the real world problem that occurs when everybody in an enterprise gets a new model of portable telephone. Most of the competencies in using a portable phone don’t change, but there is a definite competency problem where newfangled features of this new particular phone model are concerned. With any luck, the phone vendor can provide a training guide that can be used to quickly define the knowledge and skills that will be required. The phone vendor might even provide a standard competency model tree and associated RCDs, along with the job aid and training resources that can support or build the competencies defined in those RCDs. If not, the framework approach makes it easy to analyze the problem and turn it into an operative solution very quickly.

For maximum flexibility, competency trees that can be extended by referencing other trees, or merged into larger ad-hoc trees are more practical than massive, all-encompassing trees.

The role of ISD

Don asks Vo, an instructional designer in the training department, to interpret the directive and specify the competencies that must be mastered by employees in order to comply. Vo comes up with a small hierarchical competency model that includes an outline and four very specific competency definitions.

Vo was able to search the competency definitions and models repository to find and reuse several competency definitions that had already been defined as the terminal objectives for short courses on standard reports and operational evaluations, so it took him only a couple of hours. Don takes the competency model he got from Vo to the Operations office, where people are pleased to find out that most employees already master most of the competencies defined in this model. Don adds this model to the organization’s official competency model for handling emergencies and publishes the updated model.

Instructional System Design methodology (ISD), has been required by policy in many military, government and corporate environments. In many cases, the analysis phase of ISD produces a lot of paper but little that converted into actionable training content. This framework provides a way to formally capture results of ISD analysis in the form of RCDs and competency maps. The language that specifies a Terminal Learning Objective (TLO) or an Enabling learning objective can be captured as formal statements in the definition part of a RCD. The RCD standard data model can capture a learning objective regardless of the level of granularity of that objective in any particular context.

Don asks Vo to prepare a quick assessment and a simple learning resource for each of the new competencies, and to publish them to the organization’s learning resources repository. The most difficult assessments and learning resources already existed, because they had been built to match the competencies that had already been defined and that were reused here. Vo reviews those existing assessments and resources for to make sure they make sense in this new context. He finds that one of the assignments is a little too context specific, modifies it to be relevant also if it used in this new context of office emergency, and submits an updated version of the assignment for review and publishing to the repository.

Two days after the new directive was published, a group of new hires begins their intake training. Their training objectives automatically include the competencies required to implement new directive. Meanwhile, employees have already been notified by the competency management system that the training is available to learn how to meet the requirements of the new directive.

The hierarchical map approach to competency modeling allows breaking away from the tyranny of the TLO and ELO nomenclature where one wants to represent coarser and finer levels of competency than is represented in this model. On the other hand, it also allows communities of practice to specify the nomenclature they prefer. If a community of practice wants to use it in more rigid ways, it can be done without breaking conformity with the framework. In other words, you can apply any policy you like and the framework still works.

Job competency profile

The competency profile for a job or occupation can be specified using the proposed framework. The job competency profile is basically a competency model which consists of

· A collection of RCDs

· A competency map (tree) that references the RCDs. By default, the children of any node in the map must be satisfied to satisfy the parent node. For example, let us say that node A in the map A has children X, Y and Z. The structure of the competency map indicates that X, Y and Z should be rolled up to add up to A. In other words, competency in X, Y and Z is required to achieve competency in A.

· Optionally, acceptable proficiency levels that are attached to specific nodes in the competency map. If no level is specified, the proficiency is assumed to be “100%”.

· Optionally, rules, such as relative weights, that are attached to specific nodes in the competency map to override the default rollup of competency assumptions.

A typical user interface as described above can be used to explore a job competency profile, with a tree in the left pane and a display or work area in the right pane.

Capturing individual or group competency data

Chris is impressed by Don’s competency model. It looks like anyone who masters this will be able to survive and handle just about any office emergency while saving the company money by sticking to the company policies. But clearly not everybody is quite ready. Before investing in training, Chris asks Don to investigate which competencies already exist among the staff, and which are critically missing. In other words, Don must do what is commonly called a skill gap analysis. But first, Don must come up with an assessment plan. Good idea, says Chris, but please keeps the costs down. Don’t forget that we already have data from the intake training for new hires, and from the evaluations filled in by the floor wardens when we did our last annual fire drill. See if you can use that in your readiness assessment.

Competency data for individuals or groups may be used for a variety of business or operational purposes:

· Readiness assessments

· Recruitment

· Performance evaluations

· Skill gap analysis

· Personalized training plans

· Operational planning

· Adaptive sequencing in training courses

· Just in time workflow support through targeted job aids

· Assembling or updating a personal portfolio

The framework facilitates all these processes using simple data models and processes, and by enabling the services that can exchange the data in predictable ways.

Data about the competencies of an individual or group may come from different sources:

· Existing records in more or less standardized forms (e.g. HR databases, resumes, transcripts, annual review notes, certificates of achievement, job performance records, etc.)

· Porfolios

· Assessment results embedded in tracking data from online learning (e.g. SCORM tracking data)

· Evidence about an equivalent competency. For example, if you know how to plug in a toaster, there is a very high probability that you also know how to plug in a blender.

· Evidence about a related competency that includes the target competency. For example, if you know how to plug in a toaster, it can be safely assumed that you can recognize a power plug and outlet.

· Various kinds of assessments

The framework facilitates the capture and exploitation of competency data from all those sources.

The simple competency modeling in this framework, using reusable competency maps, also helps in some situations where there is data but it does not map exactly. For example, it cannot be safely assumed that if you know how to plug in a toaster you also know how to diagnose what is wrong if the outlet has no power.

Figure 12 - Capturing existing competency data into the framework

Evidence of competency

Evidence of competency is usually based on some form of assessment. Assessments also come in many different flavors. For the purpose of this paper, we will use a very general definition of assessment:

An assessment is any form of judgment, automated or not, that produces results that that can be captured in a digital data record.

For example, if a reviewer inspects a claim of competency in a person's resume and decides that it looks credible, this is a form of assessment for which a result may be captured in an evidence record. Assessments may be performed by humans or automated. They can be simple or multi-faceted. This paper proposes a common model to distill assessment results into evidence records, regardless of the instrument or form of assessment.

Obviously, different forms of assessments merit different levels of confidence. The results of a properly conducted 360 degree assessment are clearly more credible than those from a SCORM learning object that was used on an unsecured computer somewhere on the Internet.

The model proposed in the framework assumes that there can be at least one form of assessment for every defined competency. In practice, assessments often evaluate multiple competencies, or evaluate composite competencies. We assume that each of those competencies, component competencies or facets of competency can be represented by a reusable competency definition, possibly with some additional context-specific metrics. For example, an assessment may be designed or specified to evaluate whether a racing pit team is capable of changing a race car tire in less than 30 seconds. Another assessment may be required to assess whether a person achieves a set level of proficiency.

Distilling competency evidence

The distillation process for a single competency or competency facet is summarized in Figure 1. This distillation process is used for each reusable competency definition for which one wants create a corresponding competency record.

1. Some form of assessment of a person or team's proficiency in a specific competency or competency facet produces results. Different forms of assessment produce different kinds of results. There is no single standard for assessment results.

2. The assessment results are distilled into standard evidence records. Results may come in at different times from different sources.

3. The most credible evidence record is used to create or update a competency record that states whether an individual or team satisfies the requirements for the competency, and at which proficiency level.

Figure 13 - Summary of the competency evidence distillation process

Standard evidence records and a competency record can also be qualified with a confidence rating. This confidence rating is determined by policies. The proposed standards do not specify what those policies are, since they will differ with enterprises and communities of practice. However the proposed model provides a way to capture the result of applying those policies, and to update the data when policies change.

The proposed framework does not specify where personal competency records are stored. Typically, they may be stored in a personal portfolio, according to one of the many contending specifications for a “personal portfolio” or “ePortfolio” or “learner portfolio”.

Figure 14 - Portfolio vs. standard competency records

Competency records

A competency record states that:

· A person (or a team)

· Has a known competency status, which may be qualified with a proficiency level, regarding a particular competency or facet defined in a reusable competency definition

· This determination is based on one or more records of evidence, which can be looked up

· Based on enterprise policy, the evidence is considered more or less credible

Depending on enterprise policy, the competency record is updated when new evidence records are considered, or when the confidence accorded to one or more forms of evidence changes.

For example, one enterprise's policy might state that competency records are updated on an annual basis, based on the evidence collected during the year. Another enterprise policy might state that the competency records should be updated on an ongoing basis, as soon as new evidence becomes available.

Figure 15 - Competency record

Competency evidence

A competency record must be supported by evidence. There may be one or many evidence records for the same competency. All these records reference the same competency definition and all the evidence records have the same structure, regardless of the actual source of evidence.

A competency evidence record states that:

· A person (or team) has a known proficiency in a particular competency defined in a reusable competency definition

· The evidence that supports this determination is available in the form of assessment results that can be looked up

· The results of a particular assessment constitute the evidence, and have been distilled into a proficiency status and score.

Based on enterprise policy, different confidence ratings may be associated with different sources of evidence and types of evidence. Typically, the source will be some form of assessment, but a distillation policy may also allow direct creation of an evidence record through an "on the fly" assessment of some other kind of source.

Figure 16 - Evidence record

Source of competency evidence

A source of competency evidence

· Is a tangible document, digital or not, that can be used as the source for an evidence record.

· Can be referenced in a persistent way in an evidence record (e.g. through some kind of locator or URI)

· Can take many forms, e.g. a certificate, a transcript, a driver’s license, a note inserted in a personnel record, a set of SCORM runtime tracking data according to IEEE 1484.11.1, a portfolio, etc.

· May consist of, or contain formal assessment results, but not necessarily

· Assessment results may conform to a standard, or be easily distilled to a standard format.

· Is typically retained in some secure storage to allow auditing.

· May have an expiration or destruction date.

· &nbs