Competency Data for
Training Automation
White Paper – Draft 0.2 - 12 May 2005
Document status: Draft
Abstract
Competitive performance in today’s organization requires a good handle on how to acquire, recognize and use competencies within the organizations. Automated competency tracking and management in the context of performance support, training and adaptive online learning requires a systematic way to define and track competencies for individuals and teams. However, the competency data may come from a variety of sources and in many different formats. This paper proposes a simplified framework that uses simple, standard data formats to help automate the collection and adaptive assessment of individual and group competencies. The framework also supports the automation of skill gap analysis as well as the automation of adaptive performance support. The framework supports not only the automation of various processes, but also well informed decisions by humans. By keeping the data and processes as simple and transparent as possible, humans can apply common sense, priorities and policies with a full understanding of what is going on. At the same time, the framework supports a variety of enterprise policies, as well as credibility issues, sanity measures to avoid corruption by unreliable data, auditing, and general security, confidentiality and privacy requirements. The main goal, however, is to enable this functionality with the simplest possible model while taking advantage of standards that already exist. This white paper may result in specific proposals for new standards or for inclusion in existing standards projects.
Contents
Framework application scenarios
Capturing competency requirements
Introducing the Reusable Competency Definition (RCD)
Introducing the Reusable Competency Map (RCM)
Capturing individual or group competency data
Distilling competency evidence
Data standards from learning technology
Reusing or defining competency definitions
Security, confidentiality and privacy
Targeted assessments and quizzes
Targeted, adaptive learning plans and packages
Targeted learning plans and packages
Adaptive learning plans and packages
Automated and semi-automated learning packages
Dealing with unavoidable changes
Data for analytics to support process changes
Figures
Figure 1 - Finding reusable competency definitions or competency models
Figure 2 - Sources of competency requirements
Figure 3 – Capturing competency requirements into standard data formats
Figure 4 – Outline view of a sample competency tree fragment
Figure 5 – RCDs are intended to capture only the reusable portion of competency data
Figure 6 – The standard parts of a reusable competency definition
Figure 7 - Reusable competency definitions don't have to be reinvented for each context
Figure 8 - Example of advanced search form for RCDs or competency models
Figure 9 - Referencing RCDs in a competency map
Figure 10 – Different competency maps representing different models of the same competency
Figure 11 - Task models can be simple hierarchies of tasks or very complex models
Figure 12 - Capturing existing competency data into the framework
Figure 13 - Summary of the competency evidence distillation process
Figure 14 - Portfolio vs. standard competency records
Figure 17 – Sources of evidence
Figure 18 – Assessment results
Figure 19 - Enforcing sanity in handling competency evidence from SCORM
Figure 20 – Distilling competency information from the interview process
Figure 21 – Distilling a single competency record from multiple evidence records
Figure 22 - Table summarizing a sample confidence policy
Figure 23 - Assessment request
Figure 24 - From personalized assessment plans to competentency records
Figure 25 - Assessment request data model
Figure 26 - Assessment result data model
Figure 27 - Combining repositories, competency records and personalized assessments
Figure 28 - Developing learning content
Figure 29 - Skill gap analysis
Figure 30 – An integrated competency management, assessment and learning framework
Figure 31 - Performance support
Figure 32 – Putting it all together: Simplified framework exploiting standard competency data
Introduction
Scope
This paper proposes a simplified framework that uses simple, standard data formats to help automate the collection and adaptive assessment of individual and group competencies. The framework also supports the automation of skill gap analysis as well as the automation of adaptive performance support. The framework supports not only the automation of various processes, but also well informed decisions by humans.
In scope:
· Basic data models for the representation of competency definitions, simple competency models, and personal competency records.
· Basic data models for some types of assessment data.
· Simple hierarchical competency models.
· Metadata associated with various types of data instances to allow automatic or manual searches and some automated processes.
· Basic automation processes for the collection, distillation and storage of competency data for individuals and groups.
· Basic automation processes that exploit competency data, such as competency profiles, readiness reports, skill gap analysis, personalized assessments, and personalized performance support.
· Personalized, adaptive training based on performance requirements and personal profiles.
Out of scope but assumed to exist
· Processes by which real world requirements expressed as operational objectives, policy directives, business process documents and technical documentations are massaged into performance specifications and competency definitions.
· The actual construction and maintenance of task models, ontologies, complex competency models. The framework assumes that those can be stored and exploited, but does not specify how to construct or maintained them.
· Specific storage technology, repository configuration, or network topologies.
· Personal portfolios and other personal profiles, such as would be maintained by a human resources department or in school registration records.
· Specific security infrastructure, processes and policies. The framework is however designed to facilitate the application of security, confidentiality and data sanity policies.
Out of scope and not required by the framework
· Artificial intelligence and related data collection and processes.
· Massive, all-encompassing and immutable competency models
Purpose
In organizations and enterprises that require many people to be ready to perform a variety of more or less specialized tasks in order to fulfill the objectives of the enterprise, the quality of training, as well as the costs and time sunk into training, have major impact on performance. Competency assessments, whether it is to find right people or teams for the job or to determine training needs, is difficult without relevant and usable data. Effective training and performance for large numbers of people is a huge problem that can, at least in theory, benefit greatly from automation, if only by ensuring that the correct training is targeted and that no time is wasted on unnecessary training.
Although the requirements in education or in executive development programs have different names, in effect they are very similar. Achieving verifiable learning outcomes, domain-specific competencies, and general competency as a well rounded human being are among the top stated goals of educational institutions and human development programs. This must be done with ever shrinking budgets in the face of mounting expenses. At the same time, it is more necessary than ever to be able to demonstrate performance in the learning enterprise, in order to secure ongoing funding. Also, the outcomes to achieve must be expressed in a form that requires all participants to understand what is going on so that they can keep the humanity at the core of increasingly automated processes.
The purpose of the proposed framework is to provide the benefits of automation in support of these requirements, without falling into the trap of excessive complexity or requiring wholesale and immediate conversion of existing data and processes. The framework must also provide for meaningful human oversight over the processes, and the application of policies that may vary greatly among communities of practice.
How to read this document
Although the complete framework may appear to be complex, because of the number of different processes it supports, in practice its parts and processes are simple, and no more complex than what people are faced with in normal business, training or education today. In some cases they are simpler than some current models, which tend to be accretions of features and mismatched data.
The data and processes defined in the framework are designed to remain understandable to normal human beings. Therefore, it is recommended that the readers of this document focus on specific scenarios, one at a time, rather than trying to understand the whole framework immediately.
Acknowledgements
This paper draws on the wisdom, ideas, examples, insights and challenges from many people over the course of several years. Without them this proposed framework would not exist. Some people may not have realized how helpful even a brief remark or comment was. Others contributed major work that was seminal in various aspects of this project. A proper bibliography would far exceed in length the paper itself, but would still not capture how conceptual connections were born from the clash and intercourse of ideas in many, many books, articles, conversations, exchanges, demonstrations and encounters with a wide variety of people in schools, meetings, conventions, offices, airplanes, trains, laboratories, work sites and factory floors. Many names come to mind, and I must beg forgiveness for not remembering all of them, so I’d better stay out of trouble by citing none. To all of you, I am humbly grateful.
Reality checks
This paper is grounded in reality. It is also much easier to understand through examples, in the context of some real world scenario. Wherever possible, concrete examples illustrate concepts. The framework is designed to deal with very large automation projects as well as small ones; many of the examples in this paper will be based on the following scenario:
Corporate policy, safety regulations and insurance coverage conditions require that employees know what to do in case of an emergency at the office. Such emergencies include natural disasters, fire, sprinkler malfunction, injuries, health emergencies, workplace violence and terrorist actions.
It is therefore a competency requirement that all employees be ready to behave in any workplace emergency situation according to company policies and procedures.
General requirements
Need for performance
Most enterprises have some common basic drivers. One of them is a need for performance—the ability of the employees or staff to achieve the business objectives of the enterprise. Another one is cost containment. The business objectives must be achieved in the most economical way possible. A third is motivation. Most managers recognize that a well informed, well motivated, confident work force is more likely to produce good results.
What then is performance? It not just the achievement of an objective, it is the ability to do so without excessive cost or effort. One useful definition of performance is as the successful application of the competence of individuals and teams in the completion of a task or the pursuit of an objective.
Problems
To date much of the automation in training and performance has been fairly ineffective, or too expensive to achieve. Typical problems include:
· Poor, incomplete or belated alignment of the training with evolving business requirements.
· Overly complex and rigid systems that become so complicated that no one understands all their parts, and that any improvement is inordinately expensive.
· Inability to capture existing processes and legacy data.
· Disappointing results from promising technologies, such as artificial intelligence. In reality, the knowledge capture process required to support artificial intelligence has turned out to be much harder than anyone expected, and this often limits the scale of projects.
· Dependency on “perfect” data with full referential integrity, when in reality the available data is often less than perfect or may be mismatched, and processing accidents do happen.
· Lack of support for auditing and robust security
· Lack of support for recovery from process errors, bad data or human cheating.
· Ever shrinking funding in the face of ever increasing demands for performance and ever more complex competencies to support performance.
· Secrecy and “Not Invented Here” attitudes leading to constrained functional and data silos that resist integration with other systems.
· Lack of interoperability standards, or lack of awareness of the standards, leading to more or less absolute dependency on particular vendors or on proprietary implementations.
Doing more with less
This paper proposes a framework to support automation. Automation supports decisions by humans. By keeping the data and processes as simple and transparent as possible, humans can apply common sense, priorities and policies with a full understanding of what is going on. At the same time, the framework supports auditing and the automated application of policies to help ensure consistency in the data and in the way the framework is applied.
Doing more with less complicated models and implementations is a driving design principle of this framework. Of course, actual implementations may add value by providing better tools, better interfaces, and especially by providing content in the form of competency definitions, competency models, learning content, and so on.
For each of the problems cited above, the framework proposes a solution or at least, within the constraints of feasibility, an approach to a solution.
|
Problem |
Framework solution |
|
· Poor, incomplete or belated alignment of the training with evolving business requirements. |
Capture definitions of the enabling competencies required to achieve performance outcomes. Automation of adaptive training based on skill gaps that may be evaluated in real time according to changing needs. Automation of the specification of required training or performance support materials. Automatic matching of training resources to the competencies to be developed. Support for training resources that can evolve from “good enough” to “great”, depending on time and resources available. Support for different types of learning resources or for choices of training methods. Support for human oversight and override of any automated part of the process. |
|
· Overly complex and rigid systems that become so complicated that no one understands all their parts, and that any improvement is inordinately expensive. |
Support a service oriented architecture (SOA) built around business processes and services rather than around a monolithic core. Simple data models that can be combined in powerful ways. Simple “pipeline” and algorithmic processes that can be combined in powerful ways. Small number of predefined service interfaces to reduce the number and complexity of security and privacy filters. Clear definition of processes subject to the application of customized policies, e.g. what kind of competency evidence to trust. Straightforward data stores that can be implemented using generic industry standard technologies Data stores through catalogs that can be presented in a form familiar to any user of commercial web sites like Amazon or eBay, and searched easily using familiar search interfaces. |
|
· Inability to capture existing processes and legacy data. |
Provisions for a “distillation” process to distill legacy data into standard data that supports the automation features of the framework. Provisions to retain audit trails and access to the rich data that are not preserved in the distillation results. Resilient design that can function with incomplete data or with data of variable trustworthiness. |
|
· Disappointing results from promising technologies, such as artificial intelligence. In reality, the knowledge capture process required to support artificial intelligence has turned out to be much harder than anyone expected, and this often limits the scale of projects. |
No dependency on artificial intelligence. Note that artificial intelligence can be used to mine data and extend the framework in various useful operational ways. |
|
· Dependency on “perfect” data with full referential integrity, when in reality the available data is often less than perfect or may be mismatched, and processing accidents do happen. |
Resilient design that can function with incomplete data or with data of variable trustworthiness. “Best effort” design that anticipates data quality problems and simplifies data models to allow easy human detection and auditable correction of errors. |
|
· Lack of support for auditing and robust security |
Support for full audit trails for any data created or modified in the framework. Preservation of audit trails in automated processes. Allow auditing of arbitrary decisions, such as competency equivalency decisions. Small number of service interfaces to reduce the number and complexity of security and privacy filters. Clear definition of processes subject to the application of customized policies, e.g. what kind of competency evidence to trust. Supports closed-circuit implementations (e.g. offline, closed intranet) as well as highly distributed implementations. Implementations may be federated trough narrowly defined, easy to secure “pipes”. |
|
· Lack of support for recovery from process errors, bad data or human cheating. |
Allow automated after the fact corrections of data records that may be influenced by bad data introduced in the system. Provides for after the fact correction of results influenced by cheating. |
|
· Ever shrinking funding in the face of ever increasing demands for performance and ever more complex competencies to support performance. |
Highly scalable using the same models; no need to overbuild to get benefits from the framework. Data and processes can be migrated to implementations of any size without data loss. Simple data models that can be combined in powerful ways. Simple “pipeline” and algorithmic processes that can be combined in powerful ways. Small number of service interfaces to reduce the number and complexity of security and privacy filters. Clear definition of processes subject to the application of customized policies, e.g. what kind of competency evidence to trust. Straightforward data stores that can be implemented using generic industry standard technologies. |
|
· Secrecy and “Not Invented Here” attitudes leading to constrained functional and data silos that resist integration with other systems. |
Support for a variety of policies specific to a community of practice. The policy implementations can remain highly customized to support special requirements. Reduce design costs for generic functionality. |
|
· Lack of interoperability standards, or lack of awareness of the standards, leading to more or less absolute dependency on particular vendors or on proprietary implementations. |
The framework is based on existing, emerging or proposed standards. The design is largely independent of any implementation or extension. The standard features of the core data models and processes remain functional without proprietary extensions. The framework does not depend on a specific vendor or technology platform (e.g. Microsoft, Sun or *nix). |
Framework application scenarios
To facilitate understanding, the framework will be described in the context of major application scenarios. These major applications are:
· Capturing competency requirements(page 11)
· Capturing individual or group competency data (page 25)
· Targeted and adaptive assessments (page 41)
· Targeted, adaptive training and learning (page 46)
· Competency-driven adaptive performance support (page 49)
Capturing competency requirements
Competencies in the example competency scenario
The introduction stated that many examples would be based on the following scenario:
Corporate policy, safety regulations and insurance coverage require that employees know what to do in case of an emergency at the office. Such emergencies include natural disasters, fire, sprinkler malfunction, injuries, health emergencies, workplace violence and terrorist actions.
It is therefore a competency requirement that all employees be ready to behave in any workplace emergency situation according to company policies and procedures.
Firs, we will look at how the framework allows us to capture the detailed competency requirements to support the scenario.
Defining competencies
Chris, compliance manager, asks Don to put together a plan to bring all employees and new hires to a reasonable level of competency in handling emergencies at the office. After some investigation, Don realizes that no one in the organization really knows what this really means.
Don turns to a competency management system. This competency management system provides access to repositories of published of reusable competency definitions and competency models. Using a search similar to Google’s advanced search, he uses a simple form to look for existing competency definitions, using the keywords “office emergency”. He also checks the option to find competency definitions for which a competency model with more detail exists.
This first search returns a list of competency definitions published by various entities. Some of them are free; some of them require a payment to see in detail. All of them have an associated competency model, but the prices for the competency models range from free to hundreds of thousands of dollars. Don chooses to order his search results by price of the competency model, and chooses one that looks like it was published by a reputable source.
Figure 1 - Finding reusable competency definitions or competency models
Competencies are a very vast subject complicated by very strong opinions and cultural traditions. High level competencies are highly personal and essentially impossible to define and describe—the sum is greater than the parts. For example, the competency of a charismatic leader can be analyzed, but such an analysis will never capture some intangible aspects of what makes this person capable of such performance. However there is a lot about competency information and competency data that can be described or captured to enable systematic performance improvement, performance support and training. Standard data models and clearly defined processes or workflows can then enable various forms of automation, such as computer storage of personal competency profiles, skill gap analysis, individualized performance support and training, and decision support for individuals and their managers. For example, such data can be used to find learning resources designed to build the specific competencies required for a task.
The real world as source of competency requirements
Figure 2 - Sources of competency requirements
Normally, competency requirements are drawn from more or less explicit real world requirements. The processes used to analyze those requirements vary from simple head-scratching and pure guesswork to very elaborate, highly validated workflows that result in complex semantic modeling of the problem space. This framework does not attempt to define what such a process it, or what the process involves. However, this framework assumes that it is possible to distill the results of the process into data that can be captured using standards-based data formats and models, and that distillation can be an ongoing process as things evolve.
The distilled data used in this framework may not be as complete as more elaborate descriptions of the real world, but they should be sufficient to support a number of useful scenarios at a reasonable cost. They should also make it easy to deal with inevitable changes over time.
Figure 3 – Capturing competency requirements into standard data formats
For example, analyzing the competencies required for the example scenario might first result in a hierarchy of component competencies, as shown in Figure 4.
Can handle workplace emergency
Knows and understand general emergency policies
Knows what constitutes an emergency
Knows who is responsible in an emergency
Can evacuate safely and follow proper procedures
Knows the evacuation routes
Evacuation procedures
Observe proper priorities in emergency situation
Can cite the order of priorities
Demonstrates understanding that personal safety comes first in emergencies
Demonstrates attention to ensure safety of other people in emergencies
Knows what to do to protect confidential information in emergencies
Can apply emergency property stop loss measures
Can handle a fire event
Demonstrates understanding of fire situations
Knows what to do if you detect or find a fire
Knows what to do if there is a fire elsewhere in the building
Can evacuate safely and follow proper procedures
Knows what to do immediately after a fire that caused damage
Can handle a survivable earthquake event
Demonstrates understanding of earthquake situations
Knows what to do during an earthquake
Knows what to do immediately after an earthquake
Can evacuate safely and follow proper procedures
Knows what to do immediately after an earthquake that caused damage
etc., etc., etc.
Figure 4 – Outline view of a sample competency tree fragment
Figure 4 is a fragment of a competency model in the shape of a tree or hierarchy. There are other ways to model competencies, such as a semantic network or ontology. However, when you take a small subset of a domain a tree like this one is often sufficient. Higher level competencies can be decomposed into competency facets—knowledge, behavior, skill—and more specific sub-competencies.
The tree can of course be extended into more detail, down to specific skill and knowledge definitions, such as “recovering and packaging paper documents damaged by water”.
Note that this tree is not a true taxonomy, because the same competency appears in more than one place in the tree. In this example, knowing the evacuation routes and the proper evacuation behavior are competencies that appear again and again in the collection of competencies required to handle different kinds of emergencies.
Each of the competencies listed in the tree can be defined in more detail. This can take the form of some extended description, statements of knowledge such as “can cite the name and phone extension of the current floor warden”, or some statement of performance like “given a fire somewhere else in the building, follow proper procedure for immediate safe evacuation and regrouping at a predefined meeting point without adding to danger to self or others.”
As we saw above, the same competencies may appear in more than one place in the competency tree. Thus, it makes sense to capture the definitions of those competencies in some reusable form, so they have to be defined only once.
Introducing the Reusable Competency Definition (RCD)
A reusable competency definition (RCD) captures the part of competency information that may be reused for more than one person in one or more contexts and possibly with different metrics.
Figure 5 – RCDs are intended to capture only the reusable portion of competency data
A reusable competency definition (RCD) may define a skill, knowledge, aptitude, or a learning objective. The definition may be for a facet of competency (e.g. affective, psycho-motor, cognitive facets) or for a component competency of a larger competency (e.g. “Knowing the evacuation routes” is part of the larger competency “Able to handle office emergencies”).
The making of a Reusable Competency Definition
Each RCD has at least two parts: An identifier that makes it globally unique and the "payload" that contains the actual definition information. The identifier is intended for machines. The payload is intended for humans. The payload includes data like title, description, formal definition(s) according to one or more community-specific models, and intrinsic metadata about the definition itself, such as when it was created, and so on.
Figure 6 – The standard parts of a reusable competency definition
A RCD must have at least an identifier and a title. The globally unique identifier of a RCD works like the globally unique ISBN number of a book. For most automation scenarios, you only need the identifier. Using an identifier rather than the title or content of the RCD removes any ambiguity when you refer to a particular competency definition in automatic operations. Book lists typically include ISBN numbers, one purpose of which is to avoid confusion between similar sounding titles or different editions of the same work. In the same vein, an inventory of competencies in a personal profile might use the RCD identifiers, so that there is no confusion between that might arise from similar titles.
As for a book, once a RCD is published it should not change. If a change is necessary, this is in effect a new version and the new version requires a new identifier, just like publishing a new version of a book requires a new ISBN. As with books, you may run across many copies of what looks like the same RCD; as long as the identifier is the same you know that it is exactly the same version of the same RCD.
Of course, in practice it will often be necessary to know whether a RCD has been replaced with another one. The proposed framework includes ways to do this without modifying the old RCD.
Metadata provide a means to identify the source of the definition. Metadata can also be used to capture known fixed relationship with other definitions or contexts. For example, if a new RCD replaces an obsolete RCD, the old RCD may be referenced in the metadata. This information can then be used by RCD repositories to automatically add references the new RCDs in the catalog entries they use to describe the older RCD records. There are many other use scenarios for the metadata that can be described elsewhere.
It is important to distinguish between intrinsic or embedded metadata, and external metadata. For example, if you open a book to the copyright page, you will see embedded metadata. The metadata cannot be changed without destroying the integrity of the book. But the metadata can be extracted and put in external metadata records, for example in a catalog. Unlike the embedded metadata, the external metadata records may contain additional, changeable metadata. For example, external metadata may include user reviews and references to new editions of the book. There may be different external metadata records describing the same RCD, just like there may be different external metadata records describing the same learning object. The embedded metadata never changes. The external metadata may change depending on who is describing the RCD and for what purpose.
Since catalogs are collections of external metadata, when a RCD becomes obsolete, repository catalogs can point to the new RCD without modifying the existing RCD or its embedded metadata. The addition of the reference to the new RCD only affects the external metadata captured in the catalogs. If RCD identifiers are handles, as defined in the Handle system (see http://handle.net ) or the handle-based CORDRA specification (see http://cordra.net), the handle itself can be used to find a reference to the current version. How to do this with CORDRA is not entirely defined yet, but is likely to be defined in the months to come.
New competency definitions may be created directly using the RCD standard data model. Many existing competency definitions that exist in various competency models can also be captured into that standard data format. In some cases the existing competency definition may have to broken into several RCDs . The RCD in itself is not sufficient, nor is it intended, to capture the complexity of a competency model. Rather, it is intended as a basic building block to define a competency or facet of competency at any level of granularity in a competency model.
The RCDs data model is defined by existing standards. There is an IMS specification, on track to become IEEE standard, which is under consideration for a possible ISO standard.
For more information about the standards, see http://www.ieeeltsc.org/wg20Comp/ or IMS RDCEO specification at http://www.imsglobal.org.
Reusable Competency Definitions in different contexts
As Don explores the competency model he found in the repository, it becomes obvious that some of the competencies defined in the model won’t apply to his organization. However, some are generic enough to apply just about anywhere.
Figure 7 - Reusable competency definitions don't have to be reinvented for each context
The term "reusable" in “reusable competency definition” does not imply that every RCD is reusable. Rather, it implies that it allows competency definitions captures using the RCD data model to be reusable, if it makes sense. The same data model is used, regardless of the scope of reusability. For example, a particular skill definition may be extremely specific to a particular piece of equipment, with very specific performance criteria. But it can be captured and referenced using the same data model as a more generic definition, such as "able to resolve conflicts rather than escalate them". Using the same data model allows the automatic processes and the data management to be simpler.
Reusable Competency Definitions and quality
The RCD standard is a technical standard and does not specify the quality of the content of a RCD. Similarly, ISBN is a technical standard for book identifiers, and it is entirely silent about the content or subjects of the book. The usefulness of this kind of value neutral technical standard should be obvious. For example, the technical standards for email allow all kinds of messages to be created, transmitted, stored and referenced, regardless of whether the messages are business memos, love letters or spam, short notes or lengthy detailed essays.
Of course, communities of practice that create and use RCDs will have expectations of quality and will probably set their own editorial quality policies. However, one should probably not be to restrictive. After all, there is a lot of value in citing other works in documents, even if those works don’t follow the same editorial guidelines or philosophy. We all buy other people’s products even if they are not exactly what we would build ourselves if we had the time and resources to do it. Reuse sometimes involves compromises where the benefits of reusability outweigh the desire for perfection.
Reusable competency definitions as valuable intellectual property
Because it takes time and effort to create useful RCDs, this creates value. There is no reason why published RCDs might not be copyrighted, published and licensed like other intellectual property.
Finding reusable competency definitions
In order to reuse RCDs, you muse be able to find them. RCDs can be stored in repositories that, like any repository for digital object, may be standalone or federated. Typically, repository functionality includes a way to search the content. This could take the form of a catalog that contains metadata about the actual content. For example, the World Wide Web is a huge repository which can be searched by searching the Google index. When you do a Google search, you don’t search the actual web pages. Instead, you search the Google index, which is a catalog of many pages on the Web. The search results shown by Google include a fragment of the catalog data to help you decide whether what was found is what you want.
Finding RCDs can occur in many ways, such as, for example:
· Searching or browsing by title, or by title and description
· Searching by specific aspects of the definition, such as a specific statement of behavior that exists when a particular model is used in the formal definition part of the RCD,
· Searching for specific characteristics described by embedded or external metadata, such as a particular publisher or a link to related RCDs.
· Finding a RCD automatically through a reference in a competency map or model.
Figure 8 - Example of advanced search form for RCDs or competency models
Trade secrets and classified competencies
As he identifies the competencies required for the handling of office emergencies in his organization, Don stumbles upon a procedure manual with a fat label on the front that says “Confidential information – Authorized eyes only”. The manual describes procedures to shut down and restart the security system in case of emergency. Anyone reading this manual would become aware of the security measures in place in the organization, and of the vulnerabilities that might arise in an emergency. Don needs to include this in his competency model, but at the same time prevent unauthorized people from viewing the actual definitions. The competency management repository he’s using lets him do that. He logs off and logs back in as a guest user, and when he tries to browse the competency definitions that are invisible to unauthorized eyes all he finds is meaningless, opaque identifiers.
The actual competency definitions may be a trade secret, a national security secret, or may have to be kept confidential for some reason.
Many automated operations are enabled by using a standard identifier for RCDs. For example, you might make a list of competencies that are required for a mission. That list could just be a list of opaque identifiers. Unless you are authorized to look up the full RCD records in a repository where the actual RCD content resides, you would have no way to know what the competencies are about.
A personal portfolio might include competency records, which consist of a collection of RCD identifiers with associated proficiency and evidence data. Those records in the portfolios of many individuals can be matched automatically against the list of competencies required for the mission, to generate a short list of qualified individuals, without ever knowing what the competencies actually are. Similarly, a learning management system might keep track of skill gaps and learning assignments without knowing what the assignments are about, because it just juggles records containing opaque identifiers. An authorized manager or learner, however, would typically be allowed to see the actual data that can be retrieved using the opaque identifiers. For an authorized user, everything is visible. An unauthorized user would only get meaningless identifiers.
Introducing the Reusable Competency Map (RCM)
Each of the competencies or competency component in the competency model outlined in Figure 4 on page 12 can be defined in a RCD. We saw that the same competency appears more than once in the model. In our example, “knowing the evacuation routes” appears more than once in the competency model outline. To avoid unnecessary duplication, instead of being embedded directly in the hierarchical structure, the RCDs can be in a separate collection.
If the nodes in the hierarchy just reference the RCDs, but do not actually contain them, we can say that the hierarch is a map of the RCDs. Figure 9 shows how such a structure can be represented. The RCDs are shown round as target symbols, and the hierarchy of competencies is shown as a tree-like structure. The collection of RCDs might not have any particular ordering or organization, but the competency map specifies a particular way to organize a collection of RCDs into a coherent structure of related definitions.
Figure 9 - Referencing RCDs in a competency map
Each node in the model may reference a specific RCD, but some nodes might exist only as a way to group related competencies under a common title. We will see later why this can be useful, for example when gathering assessment results or making up the title for a training course.
Reusable Competency Maps
A Reusable Competency Map is a competency map that represents a particular way to define a structured competency model, or part of a model, that uses a collection of RCDs. For example, “Can evacuate safely and follow proper procedures” includes several sub-competencies and facets. The relationships between those can be specified by a competency map in the shape of a tree where the root maps to the RCD for “Can evacuate safely and follow proper procedures”. Since this competency appears several times in the office emergency handling competency model, this smaller map can then be used as a reusable building block.
Like a RCD, a reusable competency map can be published, have metadata, and so on. There is currently no published standard for such competency maps, but there is a solid proposal which underlies much of the functionality of this framework.
Defining simple competency maps
Don shows the outline of the competency model he found to Chris. Chris scans it and points out that the model does not seem to include elevators, probably because it was designed for a company in one story tall building. Also, the model puts too much emphasis on earthquakes, which are much lower on the risk ranking in the organization’s internal policy statements.
Chris asks Don to fix this before proceeding. Don decides that the easiest way to proceed is to cannibalize what he can from the generic competency model, and to create a competency model that is appropriate for his organization. Using a drag and drop interface, he quickly begins his new model by dragging the competencies he wants to keep from the existing model into the outline of his new model. He adds a node to his tree for elevator in emergency and another one for elevator safety, and prunes some earthquake topics. He searches the repository catalog for competency definitions using the keywords “elevator and emergency” and finds a useful definition for a relevant competency, along with an associated small model that includes knowledge and behavior facets of why and how to avoid being trapped in an elevator in an emergency. He drags that model into his competency model, replacing the placeholder for “elevator in emergency”. He then does a search for elevator safety, finds out that it really does not say much that is relevant to handling emergencies, and decides to delete the placeholder for elevator safety from his competency model.
A competency map is a map of RCDs. It is not made of RCDs, but rather it shows how RCDs are related. It is a structure collection of nodes that reference RCDs.
· The simplified competency map model proposed in this framework is a hierarchy of nodes that refer RCDs. The child nodes of any node can represent facets or component competencies, or both. Such maps are called “taxonomies” in some communities of practice in the fields of competency or recruitment, even though they are not necessarily true taxonomies.
· The same RCD may be referenced by more than one competency map, and/or by more than one node in a competency map (see Figure 10).
· Different communities of practice may structure hierarchies of components and/or facets of a competency in different ways. This is a reality that cannot be overcome. Embrace it, don't fight it.
· Competency map provides part of the context for a RCD. For example, it can help assess a competency in the context of related competencies.
· One or more competency maps may be specifically referenced in metadata for a RCD, using a classification element. This may even specify a "taxonpath" that specifies the path to the node that references this RCD in the taxonomy.
Figure 10 – Different competency maps representing different models of the same competency
Ontologies vs. taxonomies
Competency maps can also be represented as directed graphs. They can be semantic networks that show the often intricate relationship between competencies and their facets and component competencies. However, while very small ontologies look easy, considerable effort is required to scale up to larger models. Useful ontologies are hard to build and even harder to keep up to date. Some interesting work is in progress on building ontologies as the organization mechanism for RCDs, but it may be a while before this work comes to fruition.
Until then, practically speaking, if an ontological or semantic approach is desired, it is probably more useful to build task models as ontologies, and then use that as the blueprint from which to identify relevant competencies or competency clusters that can be described by RCDs and simple tree shaped competency maps, than to start from a theoretical ontology model of the competencies. In practice, then, competency maps should be derived from the task models rather than the other way around.
Task models
Task models come in many different flavors, according to many different communities of practice. Some are hierarchical, others are ontologies. Some define a qualification; some are specific to a domain, e.g. "Radio equipment operation". Some are very specific, e.g. "repair automobile automatic transmission model XYZ", others can be very vast and describe an occupation, e.g. "Accountant" Some are defined as workflows; others, like the US Army MOS, are defined as hierarchical layers of disparate elements.
Figure 11 - Task models can be simple hierarchies of tasks or very complex models
It does not appear possible to standardize task models across communities of practice. However, the existence of more or less explicit task models is assumed in this paper. In the sample scenarios that were examined to validate the concepts in this paper, consideration was always given to how standard data and processes can be applied to real world task or workflows.
In the sample scenario assumed by this paper, a task model would probably start from a description of a particular kind of emergency unfolds. From that, one or more task models can be built. There might be slightly different task models for the floor wardens and for normal employees, for example. By analyzing the task model, it is possible to define the competencies required to accomplish each of the tasks.
Hierarchical competency maps are best for limited domain scopes
At this point, Don decides he needs to have others review his work for sanity. He selects the heading “Observe proper priorities in emergency situation” in his competency model and chooses a menu option to generate a detailed document version of it, including the competency definitions for each level of detail, and sends it to the head of HR asking for a review. He repeats this with different headings, and sends the generated documents to other subject matter experts.
Don gets feedback from HR. There is concern that the way understanding priorities is defined in the model may confuse people, because a statement the competency definition references a policy document that does not exist in this organization. Don decides to create a new reusable competency definition and point the” understanding priorities” heading to that new reusable competency definition instead of the irrelevant one.
Reality is very complex, and in many ways a hierarchical model of competencies is an oversimplification. Hierarchical maps are very useful, because of their simplicity. But they are probably best used to represent limited competency domains, or parts of domains. Experience has shown that massive taxonomies are extremely hard to build and maintain. Perfect large competency taxonomies will probably never exist.
On the other hand, small hierarchical competency maps are easy. So keep it simple and easy. Rather than one big hierarchy, use a lot of smaller hierarchies that are easy to build, easy to review for relevance and accuracy, easy to update. The smaller competency maps can be combined in useful ad-hoc ways when needed.
There is at this point no standard defining how to build competency maps; however, there are several potential candidates, of which the simple hierarchical competency map described in this framework is one.
Why reuse competency maps
Like RCDs, useful competency maps may require some serious work, and that makes them valuable. Therefore it makes sense to consider some competency maps as reusable, and provide a standard way to capture them into standard data objects that can be stored in repositories, published and licensed.
There is another aspect of reuse, even if a competency map is just ad-hoc and not intended to be reused in different contexts. An ad-hoc competency map may be used to represent a structure for the competencies to be tested in an assessment, and this structure can then be used to guide how the results from the assessments of individual competencies can be weighted “rolled up” into a broader assessment result. To support such uses without duplicating the competency map everywhere, a competency map must be a data object that can be referenced and stored in a repository for as long as any other data depend on it.
Reusable competency map metadata
Like RCDs, reusable competency maps can include intrinsic embedded metadata and external metadata. This makes finding, storing and publishing RCMs as simple as finding RCDs or learning resources. Thanks to the metadata, a RCM can just be another kind of thing you can look for in a catalog.
Reusable competency maps for normal human beings
Tree-shaped reusable competency maps can be represented in a computer interface and in reports in the familiar form of an outline as in Figure 4. However, a computer interface would typically allow the user to “inspect” any heading of the outline to look up the corresponding RCD. Behind the scenes, this means that when the user inspects the heading, the identifier of a RCD that is attached invisibly to that heading is used to look up the actual RCD and display it to the user. A typical user interface to explore a competency model might look like the familiar two pane view, with a tree in the left pane and a display or work area in the right pane.
If a competency map is structured as a directed graph rather than a tree, to embody an ontology or semantic mapping, the representation for normal human beings is much more complicated. Whereas navigating a tree is now a familiar metaphor for most computer users, navigating a directed graph along a variety of relation links is still a daunting problem. Most demonstrations seen so far require a grasp of the conceptual foundations of semantic mapping and mental models that are still foreign to normal people. There is a lot of research going on in various places to find a solution to this problem. Until this problem is solved, it is recommended that implementations of the framework rely on tree-shaped maps only, while allowing experimentation with directed graphs.
Adapting for changes in business performance requirements
A new directive is promulgated in the organization. Every office emergency must be reported and must be followed up with an evaluation of how it was handled.
Don realizes that he needs to extend the competency model for handling office emergencies. He adds a couple of placeholders in his outline of the model, while he ponders his next move.
Flexibility is no longer a luxury. It is a basic requirement. Business requirements change, tasks change, equipment changes. These changes must be reflected in competency models used for assessment and training. The simpler the model, the easier it is to make the changes. The framework facilitates changes in several ways:
· Capturing competency requirements in RCDs and competency models
· Decoupling the RCDs and the competency map that represents the competency model structure
· Using small competency models that can be recombined, rather than some monolithic, massive competency model.
By decoupling the RCDs and the competency model structure, we are adding a lot of flexibility. For instance, the example competency model to handle office emergencies can easily be cannibalized to adapt to a different office location where the escape routes and some other local conditions are different, by pointing to different RCDs that are specific to different locales.
Using small competency models that can be recombined helps with the real world problem that occurs when everybody in an enterprise gets a new model of portable telephone. Most of the competencies in using a portable phone don’t change, but there is a definite competency problem where newfangled features of this new particular phone model are concerned. With any luck, the phone vendor can provide a training guide that can be used to quickly define the knowledge and skills that will be required. The phone vendor might even provide a standard competency model tree and associated RCDs, along with the job aid and training resources that can support or build the competencies defined in those RCDs. If not, the framework approach makes it easy to analyze the problem and turn it into an operative solution very quickly.
For maximum flexibility, competency trees that can be extended by referencing other trees, or merged into larger ad-hoc trees are more practical than massive, all-encompassing trees.
The role of ISD
Don asks Vo, an instructional designer in the training department, to interpret the directive and specify the competencies that must be mastered by employees in order to comply. Vo comes up with a small hierarchical competency model that includes an outline and four very specific competency definitions.
Vo was able to search the competency definitions and models repository to find and reuse several competency definitions that had already been defined as the terminal objectives for short courses on standard reports and operational evaluations, so it took him only a couple of hours. Don takes the competency model he got from Vo to the Operations office, where people are pleased to find out that most employees already master most of the competencies defined in this model. Don adds this model to the organization’s official competency model for handling emergencies and publishes the updated model.
Instructional System Design methodology (ISD), has been required by policy in many military, government and corporate environments. In many cases, the analysis phase of ISD produces a lot of paper but little that converted into actionable training content. This framework provides a way to formally capture results of ISD analysis in the form of RCDs and competency maps. The language that specifies a Terminal Learning Objective (TLO) or an Enabling learning objective can be captured as formal statements in the definition part of a RCD. The RCD standard data model can capture a learning objective regardless of the level of granularity of that objective in any particular context.
Don asks Vo to prepare a quick assessment and a simple learning resource for each of the new competencies, and to publish them to the organization’s learning resources repository. The most difficult assessments and learning resources already existed, because they had been built to match the competencies that had already been defined and that were reused here. Vo reviews those existing assessments and resources for to make sure they make sense in this new context. He finds that one of the assignments is a little too context specific, modifies it to be relevant also if it used in this new context of office emergency, and submits an updated version of the assignment for review and publishing to the repository.
Two days after the new directive was published, a group of new hires begins their intake training. Their training objectives automatically include the competencies required to implement new directive. Meanwhile, employees have already been notified by the competency management system that the training is available to learn how to meet the requirements of the new directive.
The hierarchical map approach to competency modeling allows breaking away from the tyranny of the TLO and ELO nomenclature where one wants to represent coarser and finer levels of competency than is represented in this model. On the other hand, it also allows communities of practice to specify the nomenclature they prefer. If a community of practice wants to use it in more rigid ways, it can be done without breaking conformity with the framework. In other words, you can apply any policy you like and the framework still works.
Job competency profile
The competency profile for a job or occupation can be specified using the proposed framework. The job competency profile is basically a competency model which consists of
· A collection of RCDs
· A competency map (tree) that references the RCDs. By default, the children of any node in the map must be satisfied to satisfy the parent node. For example, let us say that node A in the map A has children X, Y and Z. The structure of the competency map indicates that X, Y and Z should be rolled up to add up to A. In other words, competency in X, Y and Z is required to achieve competency in A.
· Optionally, acceptable proficiency levels that are attached to specific nodes in the competency map. If no level is specified, the proficiency is assumed to be “100%”.
· Optionally, rules, such as relative weights, that are attached to specific nodes in the competency map to override the default rollup of competency assumptions.
A typical user interface as described above can be used to explore a job competency profile, with a tree in the left pane and a display or work area in the right pane.
Capturing individual or group competency data
Chris is impressed by Don’s competency model. It looks like anyone who masters this will be able to survive and handle just about any office emergency while saving the company money by sticking to the company policies. But clearly not everybody is quite ready. Before investing in training, Chris asks Don to investigate which competencies already exist among the staff, and which are critically missing. In other words, Don must do what is commonly called a skill gap analysis. But first, Don must come up with an assessment plan. Good idea, says Chris, but please keeps the costs down. Don’t forget that we already have data from the intake training for new hires, and from the evaluations filled in by the floor wardens when we did our last annual fire drill. See if you can use that in your readiness assessment.
Competency data for individuals or groups may be used for a variety of business or operational purposes:
· Readiness assessments
· Recruitment
· Performance evaluations
· Skill gap analysis
· Personalized training plans
· Operational planning
· Adaptive sequencing in training courses
· Just in time workflow support through targeted job aids
· Assembling or updating a personal portfolio
The framework facilitates all these processes using simple data models and processes, and by enabling the services that can exchange the data in predictable ways.
Data about the competencies of an individual or group may come from different sources:
· Existing records in more or less standardized forms (e.g. HR databases, resumes, transcripts, annual review notes, certificates of achievement, job performance records, etc.)
· Porfolios
· Assessment results embedded in tracking data from online learning (e.g. SCORM tracking data)
· Evidence about an equivalent competency. For example, if you know how to plug in a toaster, there is a very high probability that you also know how to plug in a blender.
· Evidence about a related competency that includes the target competency. For example, if you know how to plug in a toaster, it can be safely assumed that you can recognize a power plug and outlet.
· Various kinds of assessments
The framework facilitates the capture and exploitation of competency data from all those sources.
The simple competency modeling in this framework, using reusable competency maps, also helps in some situations where there is data but it does not map exactly. For example, it cannot be safely assumed that if you know how to plug in a toaster you also know how to diagnose what is wrong if the outlet has no power.
Figure 12 - Capturing existing competency data into the framework
Evidence of competency
Evidence of competency is usually based on some form of assessment. Assessments also come in many different flavors. For the purpose of this paper, we will use a very general definition of assessment:
An assessment is any form of judgment, automated or not, that produces results that that can be captured in a digital data record.
For example, if a reviewer inspects a claim of competency in a person's resume and decides that it looks credible, this is a form of assessment for which a result may be captured in an evidence record. Assessments may be performed by humans or automated. They can be simple or multi-faceted. This paper proposes a common model to distill assessment results into evidence records, regardless of the instrument or form of assessment.
Obviously, different forms of assessments merit different levels of confidence. The results of a properly conducted 360 degree assessment are clearly more credible than those from a SCORM learning object that was used on an unsecured computer somewhere on the Internet.
The model proposed in the framework assumes that there can be at least one form of assessment for every defined competency. In practice, assessments often evaluate multiple competencies, or evaluate composite competencies. We assume that each of those competencies, component competencies or facets of competency can be represented by a reusable competency definition, possibly with some additional context-specific metrics. For example, an assessment may be designed or specified to evaluate whether a racing pit team is capable of changing a race car tire in less than 30 seconds. Another assessment may be required to assess whether a person achieves a set level of proficiency.
Distilling competency evidence
The distillation process for a single competency or competency facet is summarized in Figure 1. This distillation process is used for each reusable competency definition for which one wants create a corresponding competency record.
1. Some form of assessment of a person or team's proficiency in a specific competency or competency facet produces results. Different forms of assessment produce different kinds of results. There is no single standard for assessment results.
2. The assessment results are distilled into standard evidence records. Results may come in at different times from different sources.
3. The most credible evidence record is used to create or update a competency record that states whether an individual or team satisfies the requirements for the competency, and at which proficiency level.
Figure 13 - Summary of the competency evidence distillation process
Standard evidence records and a competency record can also be qualified with a confidence rating. This confidence rating is determined by policies. The proposed standards do not specify what those policies are, since they will differ with enterprises and communities of practice. However the proposed model provides a way to capture the result of applying those policies, and to update the data when policies change.
The proposed framework does not specify where personal competency records are stored. Typically, they may be stored in a personal portfolio, according to one of the many contending specifications for a “personal portfolio” or “ePortfolio” or “learner portfolio”.
Figure 14 - Portfolio vs. standard competency records
Competency records
|
A competency record states that: · A person (or a team) · Has a known competency status, which may be qualified with a proficiency level, regarding a particular competency or facet defined in a reusable competency definition · This determination is based on one or more records of evidence, which can be looked up · Based on enterprise policy, the evidence is considered more or less credible Depending on enterprise policy, the competency record is updated when new evidence records are considered, or when the confidence accorded to one or more forms of evidence changes. For example, one enterprise's policy might state that competency records are updated on an annual basis, based on the evidence collected during the year. Another enterprise policy might state that the competency records should be updated on an ongoing basis, as soon as new evidence becomes available. |
Figure 15 - Competency record |
Competency evidence
A competency record must be supported by evidence. There may be one or many evidence records for the same competency. All these records reference the same competency definition and all the evidence records have the same structure, regardless of the actual source of evidence.
|
A competency evidence record states that: · A person (or team) has a known proficiency in a particular competency defined in a reusable competency definition · The evidence that supports this determination is available in the form of assessment results that can be looked up · The results of a particular assessment constitute the evidence, and have been distilled into a proficiency status and score. Based on enterprise policy, different confidence ratings may be associated with different sources of evidence and types of evidence. Typically, the source will be some form of assessment, but a distillation policy may also allow direct creation of an evidence record through an "on the fly" assessment of some other kind of source. |
Figure 16 - Evidence record |
Source of competency evidence
|
A source of competency evidence · Is a tangible document, digital or not, that can be used as the source for an evidence record. · Can be referenced in a persistent way in an evidence record (e.g. through some kind of locator or URI) · Can take many forms, e.g. a certificate, a transcript, a driver’s license, a note inserted in a personnel record, a set of SCORM runtime tracking data according to IEEE 1484.11.1, a portfolio, etc. · May consist of, or contain formal assessment results, but not necessarily · Assessment results may conform to a standard, or be easily distilled to a standard format. · Is typically retained in some secure storage to allow auditing. · May have an expiration or destruction date. · Is more or less trustworthy and credible. |
Figure 17 – Sources of evidence |
There is no point in trying to standardize sources of evidence. However, a standard vocabulary for types of sources of evidence would be useful.
How a source of evidence is distilled into one or more evidence records depends on the type of source. For example, various aspects of a portfolio might be assessed for signs of competencies defined in various competency definitions. Depending on policy requirements, the result of this may be captured in formal assessment records (see below) that can then be distilled into evidence records with a full audit trail, or directly summarized in evidence records that point back to the portfolio as the source, with a confidence rating that depends on who is creating the evidence record.
Assessment result
|
An assessment result · Is a tangible, archivable document that can be used as the source for one or more evidence records · References one or more reusable competency definitions, or can be mapped easily to one or more reusable competency definitions. · Results from one of may forms or assessment, that may be formal or informal · Is typically retained in some secure storage to allow auditing. · May have an expiration or destruction date. · Is more or less trustworthy and credible. There is no point in trying to standardize all assessment results since there are many kinds of assessments, and new ones are invented every day. However, some standards like IEEE 1484.11.1 can be used for many common cases. |
Figure 18 – Assessment results |
Different assessment instruments may be treated differently as sources of evidence. For example, the results reported by an online tutorial are typically less credible than those reported by a supervisor who can observe a person's performance of the task supported by the competency. Given the same assessment instrument, results reported by an unknown third party are less credible than those reported by an enterprise insider, and so on.
Enforcing sanity policies
The quality of assessment data is of course very important. This is why it is important to be able to assign confidence rating to any data that purports to state that someone is competent in anything. Policies must guide the determination of confidence ratings in the distillation of competency evidence. Different sources of evidence are more or less credible. A self-report in a resume is less credible than a validated, proctored exam result. An assessment by a manager is less credible than a well conducted 360 degree assessment, because the manager may have a grudge or an affective attachment to the person being assessed.
Generally speaking, one should be able to guard the content of a competency record against corruption by unreliable data. Even so, the competency record itself should be rated by how much it can be trusted. For example, one can create a competency record when the only source of evidence is a college degree, but this cannot be trusted as well as when there is evidence of actual performance.
Simple to determine the confidence ratings in the distillation process will be examined in more detail in a subsequent section of this paper.
Don has collected some impressive statistics showing that most employees in his assessment sample already know what to do in an emergency, but there are significant competence gaps. In particular, some people would probably take unnecessary risks, and stop loss measures in the aftermath of an emergency are widely misunderstood. He has cross-tabulated data showing a percentage of readiness for each competency in the model, as well an outline of the missing competencies for each of the employees tested. But when he presents his results at the next manager’s meeting, several people question his data.
How do we know that the problem is really that bad, asks Anne. Training is expensive and I don’t want to take my people away from their job any more than is absolutely necessary. It all comes down to a matter of confidence. If you can tell me that you are 70% sure that someone needs training, then we’ll do it. But if you’re not, then forget it.
Don came prepared. For each of his data points he has a confidence rating. He can explain how the confidence rating was calculated automatically for each competency record according to a simple policy he worked out with the assistant director of human resources. Clearly, many of the assessments are little more than hand waving, judging by the low confidence rating. Don explains that many of the assessments were done using a assessment instruments purchased from a publisher specializing in security topics. Those assessment instruments match a set of RCDs that were also obtained from the publisher, and the publisher certifies that they have been validated. Other assessment instruments had no serious validity or reliability studies done on them, so their results were automatically given a lower confidence rating. The assessments were delivered online without proctoring, which further lowers the confidence rating for their results. However, Don believes that, as long as one does not blindly believe them, they are still useful indicators. He suggests that more credible evidence could be collected by setting up a drill and observing some actual performance.
Sanity policy scenarios
Scenarios: Distilling results from online assessments
In a training and knowledge management context, it is very important to be able to access competency information. For example, SCORM allows the creation of adaptive learning activity packages that can skip the competencies already mastered by the learner.
On the other hand, one must exercise care when automated processes are allowed to update competency data. For example, SCORM can report status and proficiency levels for specific competencies. One would not allow the SCORM data to modify a person's competency record without at least some filtering through policy. It is not wise to allow the records that govern a person's career or promotion opportunities to be tainted by an unsupervised, online test that may or may not have been validated. It is too easy for someone to stand in for someone else in such a test. Figure 19 illustrates how competency data might flow in and out of a SCORM delivery environment.
Figure 19 - Enforcing sanity in handling competency evidence from SCORM
Scenario: Distilling results from a recruitment interview
A formal recruitment interview typically will result in various assessment notes. In the more formal cases, the interview is structured to assess specific competencies or facets of competency according to a preset outline. In less formal cases, there is still usually an attempt to address key issues relevant to specific competencies of facets of competencies. The result of the interview may be captured in various formats, ranging from scribbled notes to formal data entry in a computer-based form. An interview often follows some assessment of the applicant's resume or portfolio. That pre-assessment may have been used for screening only, or to raise flags for interview questions, and the results of the pre-assessment may be recorded along with the results from the interview and recruitment process.
After an interview there may be one or more sources of evidence for evidence records:
· Interview notes or other formal results can be matched to specific competency definitions to distill evidence records with a confidence rating that may vary depending on the seniority or qualifications of the interviewer.
· Resume statements can be matched to specific competency definitions to distill evidence records. These will typically have a low confidence rating, since the applicant's own claims should always be considered with caution.
· The applicant portfolio can be assessed by an expert for signs of mastery of various competencies. The result of this assessment can be a collection of evidence records with a confidence rating that depends on the seniority or qualifications of the assessor.
Figure 20 – Distilling competency information from the interview process
At the end of the day, if there is more than one evidence record for any competency, the record with the highest confidence rating is used to distill a competency record. The competency record will carry a confidence rating that can then be used to decide whether it can be trusted for a hiring decision. For example, in an attempt to automatically roll up the competency record data according to a competency map that represents the competency requirements for a particular job, an automated system may detect that a particular competency is not supported by sufficiently credible evidence. This in turn may trigger a follow-up activity to fill this information gap in a way that generates more credible evidence. This might be as simple as a phone call from a trusted personnel specialist to the applicant to ask a question that was missed in the interview, or verification of a claimed qualification with a university or previous employer.
Scenario: Distilling evidence from multiple sources
When data from multiple sources results in more than one evidence record for any competency definition, the record with the highest confidence rating is used to create or update a single competency record. Only evidence records that have not expired are considered, of course. If two evidence records have the same confidence rating, the most recent one is used.
Figure 21 – Distilling a single competency record from multiple evidence records
Scenario: Distilling expiration dates
Some evidence records may include an expiration date or other time limit. This can be carried over into the competency record. When the competency record expires, the existing evidence records can be searched for the record with the highest confidence rating that has not expired yet. If two evidence records have the same confidence rating, the most recent one is used.
An enterprise policy may dictate that the competency data should be updated as soon as new evidence is available. In that case, when a new evidence record is created, or an existing evidence record is updated, this can trigger the distillation into a competency record. If the confidence rating for the new evidence is higher or equal to the confidence rating of the existing competency record, the competency record is updated to use the new evidence. Otherwise the new evidence has no effect.
Assessment instruments
Assessment instruments come into many flavors, more or less formal, and more or less complex. The validity of assessment instruments is extremely variable. Some assessment instruments go through very thorough design processes and validity testing using longitudinal studies to verify whether the assessment results are reliable predictors of future performances. Other assessment instruments are designed on paper napkins in a restaurant; they might consist of a list of questions to ask in an interview. Sometimes assessment instruments are checks on learning following a presentation in a computer-based training module. Assessments may be administered in very secure, trustworthy conditions. Others may be administered online with no guarantee that the assess person is actually sitting in front of the computer.
Roughly speaking, one could classify assessments along three dimensions:
· Complexity – is this assessment for a single competency, or for a more structured collection of competencies and/or competency facets?
· Results format – at one end of the scale is whether this assessment produces results that can be captured using a standard data model, such as SCORM interactions. At the other end of the scale would be a result format that is very specific to the instrument, with no hope of mapping to a standard format
· Trustworthiness – the extremes are whether the assessment trustworthy, i.e. a formally validated assessment instrument or some assessment by a qualified and highly trusted person, or whether the results of the assessment should be treated with caution as informative data with limited reliability.
Data standards from learning technology
The learning technology community has come up with useful standard data models that allow the capture and exchange of assessment results. These results can then be used directly in an automated distillation process to produce standardized evidence records. See Appendix A for a list of relevant standards.
These standards are:
· SCORM 2004 "cmi" data model. This data model is an implementation of the IEEE 1484.11.1 standard (see below). It is used in SCORM for communication between Shareable Content Objects (SCOs) and a runtime environment. SCOs can teach, assess or both. The same SCO may assess multiple competencies by referencing RCDs as objectives.
· 1484.11.1-2004 IEEE Standard for Learning Technology--Data Model for Content to Learning Management System Communication. This standard describes a data model to support the interchange of data elements and their values between a content object and a runtime service (RTS). It is based on a current industry practice called “computer managed instruction” (CMI). The work on which this Standard is based was developed to support a client/server environment in which a learning technology system, generically called a learning management system (LMS), delivers digital content, called content objects, to learners. The data model supports learner data and preferences, interactions, objectives, content-object entry, exit, and status information, time parameters, and scores.
· SCORM 2004 packages. The Sequencing and Navigation specification in SCORM 2004 may update information about specific objectives by rolling up results from the learning activities defined in the package. The learning activities do use SCOs to collect the data. A SCORM 2004 package is not limited to learning; it can be used to specify and deliver a simple or structured assessment in any SCORM 2004 conformant delivery environment. The assessment can be adaptive and use a variety of interaction techniques to elicit useful responses.
· IMS QTI 2.0 specification. See http://imsglobal.org for details. However, as of this writing the QTI 2.0 specification does not include a standard way to summarize assessment results. This means that every QTI conformant assessment may produce different kinds of results. Until a summary results specification is available, like the one that exists for the old version of QTI, or some formal mapping recommendation is created to define a method to distill those results into standard evidence records, it will be difficult to systematically use QTI data in a simple, robust framework like the one proposed here.
As discussed above, sanity policies should apply when using standardized assessment data, depending on the source. Note that the IEEE 1484.11.1 standard was designed so that it could be used to transfer and archive assessment data in a variety of settings, not just SCORM content delivery. For example, you will find that the responses to a many paper-based or interview-based assessment instruments, as well as the results of the evaluation of those responses, can be captured in the IEEE 1484.11.1 data model.
Other data standards
There is a long tradition of development and use of assessment instruments generating more or less standardized results. Every community of practice that has such results should be able to define a method to distill those results into standard evidence records.
Assessment from existing data
Existing data in HR databases, learning management systems, and other records can be used as source of evidence. Depending on the format, an assessment instrument can be designed to capture this data. In this case the assessment does not involve the individual or team being assessed, but rather it is an assessment of the existing evidence to massage it into a form that can be used to distill evidence and provide an audit record of how this was done. Such assessment instruments might range from simple forms that are used to generate a persistent record documenting the source of data, and that require transcription by a human operator; to fully automated processes that might use web service interface to extract and massage data from existing databases. We are calling this an assessment instrument because in many case some form of judgment or evaluation may be necessary to determine whether the data is reliable and relevant.
In all cases, what matters here is that there is a way to map existing data to reusable competency definitions so that standard evidence records can be distilled from these various forms of assessment. It may necessary to create the reusable competency definitions based on the available data. For example, the facets of competency evaluated in a 360 degree assessment for people in a particular position in your enterprise might each be represented by a separate reusable competency definition.
Enforcing confidence policies
As described above, standard data models for competency evidence and competency record can be embodied in simple data structures. These data structures should include a confidence rating on a standardized scale. How the confidence rating is determined, however, is a matter of policy. It does not seem practical to standardize the policies, but it is practical to standardize the result of the policies. This takes the form of confidence ratings which enable the automated filtering and distillation of evidence.
Although it does not seem possible to define a general standard for policies, a policy that specifies the confidence rating to apply for various sources of evidence might look like the table shown in Figure 22.
|
Evidence source type |
Rating |
Comments |
|
Resume statement, unverified |
0.1 |
Self rating is better than nothing. |
|
Online assessment, no authentication |
0.2 |
Anyone might stand in for the person or team to be assessed. |
|
Online assessment, identity verified |
0.5 |
Proctoring ensured that there was no stand-in. |
|
Peer |
0.5 |
|
|
Online assessment with certified validity, subject identity verified |
0.7 |
High reliability, scientifically validated assessment. |
|
Resume statement, verified by HR |
0.7 |
|
|
Supervisor |
0.7 |
Supervisor may have personal issues clouding judgment |
|
Training instructor |
0.7 |
Performance in training situation may not be entirely reliable |
|
360 degree, unaudited |
0.75 |
|
|
360 degree, audited |
0.9 |
Audited is more credible than unaudited |
|
HR Director |
0.95 |
HR Director has authority to override |
|
Executive |
1.0 |
Overrides everything else. |
Figure 22 - Table summarizing a sample confidence policy
Competency record that results from such a distillation also includes a confidence rating. This allows automation of various decision processes, according to policy. For example, a policy might state that no promotion decision can be made based on competency records where the confidence rating is less than 70%. In some other cases, as when it is necessary to assemble a team to deal with an urgent situation, a manager might decide to use a confidence rating as low as 10% as the threshold for a first cut to identify possible team members.
Complex assessments
A complex assessment for multiple competencies, competency facets and/or component competencies can be specified in a standard way.
|
An assessment request specifies: · The identifier of a reusable competency definition · Optionally, the proficiency level at which to assess · Optionally, a reference to a competency map that specifies components and/or facets of the competency to assess · Optionally, additional requirements for the assessment If no person or group to assess is specified, then this assessment request can be reused over and over as needed. In addition, an assessment request may specify one or more acceptable assessment instruments, and might specify a context in which the instrument must be used. A target confidence rating might also be specified; but in the end it will always be the recipient of the assessment results who will decide what rating to apply when distilling the assessment results into evidence records. |
Figure 23 - Assessment request |
An assessment may return a single result, based on the roll-up of the competencies or facets evaluated in the assessment. It might also return detailed results, with a status and possibly a score for each of the competencies or facets. A competency evidence record can be distilled from each of those results, as long as the result includes a reference to a reusable competency definition. For example, an assessment for the ability to do a task may include knowledge and skill sub-competencies. An individual might have the knowledge but lack the skill, in which case the roll-up would indicate a lack of competency for the task, but the detailed evidence would indicate that only the skill needs training or remediation.
Reusing or defining competency definitions
Enterprise policies and workflows determine whether and how reusable competency definitions are created. For example, a policy might state that evidence records may be created only if a suitable reusable competency definition already exists in the enterprise's catalog or in a catalog the enterprise subscribes to. Another policy might state that HR employees may create new reusable competency definitions as they analyze various sources of competency data, such as resumes, hiring requests from managers, existing competency or task models, etc.
Ideally, existing reusable competency definitions should be used wherever they are available, rather than inventing new ones. The proposed framework does depend on this to avoid datarrhea.
Reporting and analytics
Reports
The proposed model provides a way to generate useful reports that do not require expensive customization, because of the regularity of the data structures regardless of the level or type of competencies being tracked and reported. The reusable competency definition standard and the use of simple hierarchical maps rather than massive or very complex ontologies simplify the design of comprehensive reports with useful, human readable information. For example, given a reusable competency definition identifier, the title and possibly description of the competency definition can be looked up for inclusion in the report; a hierarchical competency map provides a standard blueprint to structure the report to show component competencies and/or facets of competency and how they "roll up".
The IEEE 1484.11.1 data model is currently used in SCORM 2004 only for content object to runtime environment communication. Generalization of this data model, or of the data structures used in the model, is particularly useful for reports. This is standard data model designed to report a variety of tracking information. For example, success status for multiple objectives directly maps to success and proficiency score for RCDs. Using this model as a Rosetta Stone, reports are reusable between implementations that support this data model for assessment or learning activity summaries.
Personal competency profile
The competency profile for person or group can be displayed or reported in a standard way using the proposed framework. The personal competency profile is basically structured like a competency model and consists of
· A collection of RCDs (which may exist in a repository)
· Optionally, a competency map (tree) that references the RCDs. If such a map is not available, a table of content for the competency profile can be generated by just listing the titles of the RCD.
· A competency record for each RCD.
A typical user interface as described above can be used to explore a personal competency profile, with a tree or list of competency titles in the left pane and a display or work area in the right pane.
Useful applications may be built that use a personal competency profile and look up matching RCD references in reusable competency maps. For example, given a competency record, the RCD may be viewed in the context of a more complete competency model. The more complete competency model may in turn be used as a map to guide the automatic creation of a map from the personal profile, thus making the profile easier to navigate.
Analytics
There is now a way to support meaningful analytics because the data models are coherent and the data to support analytics are unambiguous. Without this model one is forced to try to analyze widely disparate kinds of data with wildly different semantics, or much poorer data that does not adequately capture the variety of processes and sources of competency evidence.
Real-time reporting and analytics are supported if the implementation supports distillation in real time with automatic application of the confidence policies.
Examples of analytics exploiting the proposed model:
· If reusable competency definitions and/or competency maps exist to specify the competencies required to support performance on specific tasks or in particular positions, actual performance measures for an individual or team can be correlated with competency records for that individual or team. Observed performance can be fed back into the system and distilled into standard evidence records that can be used in longitudinal studies.
· The impact of the use of learning resources or performance support resources targeted to specific competencies can be verified because of the common references to reusable competency definitions are the key to relate use and result.
· The effect of different forms of interventions can be compared. Each evidence record contains a pointer back to the raw data that result form the intervention. The effect of the intervention on different individuals or groups can be measured by comparing evidence records that can be traced to the interventions, and by excluding from the analysis the individuals for which other evidence records indicate extraneous influences.
· The confidence ratings can be used in various ways to analyze credibility factors or to filter out unreliable data.
· Using the IEEE 1484.11.1 data model where more detailed analytics are required provides a way to compare apples and apples, oranges and oranges without expensive interpretation and remappings.
Security, confidentiality and privacy
Authentication and authorization
Authentication and authorization are important to avoid corruption of competency evidence and competency records. The proposed model does not specify how to deal with such security issues. However, it simplifies the security issues by isolating the data into well defined silos with well defined propagation paths.
Intentional or accidental data corruption
The sad reality is that people cheat and that even when they don't we have to work with often incomplete or unreliable data. The proposed model facilitates the implementation of policies to minimize the impact of cheating or bad data. It specifies a simple process to distill the data, minimizing the ways to circumvent sanity checks.
A fully auditable implementation would create a persistent assessment record (which might just be a simple note) or evidence record for every transaction that might influence competency records.
The model also provides a full audit trail that enables manual and automatic corrections. A manual correction should cause the creation of a persistent evidence record that identifies the person who made the correction. A more stringent policy might require that manual corrections be submitted in the form of an assessment form that gets distilled into an evidence record with the proper confidence rating applied by policy.
Automatic corrections can be implemented because of the way the data is cross-referenced in a resilient way. For example, if it turns out that a particular instructor was inflating grades, all the evidence records based on those grades can be identified because they contain a reference to the source of evidence, and their confidence rating lowered or even set to zero. Then, all the competency records that rely on these evidence records can be corrected to fall back on the next best evidence record, if one is available.
Other complex operations can also be automated to implement new policies or competency reference models. For example if two enterprises merge, and some competency definitions of one enterprise are found to be equivalent to competency definitions of the other enterprise, all the competency records can be updated after creating evidence records that specify the determination of equivalence as the source. This provides an audit trail to explain how competency records may exist for the "new" competency definitions even though there is no direct evidence of assessment for some of the employees. Obviously, the same method could also be used if a company that has a legacy competency model acquires a competency model from a vendor and uses it to remap their existing competency records into the new competency model.
Confidentiality and topic security issues
The actual competency definitions may be a trade secret, a national security secret, or may have to be kept confidential for some reason. The same applies to the assessments and assessment results. The proposed model allows management of competency records and exchange of competency information by means of opaque identifiers, without revealing the actual definitions or assessment instruments or assessment results. For example, in this model an evidence record contains no data that describes a topic or assessment data. It only contains generic values and identifiers that are useless unless you are authorized to look up what the identifiers represent.
There is of course a limit to this, which is that if an identifier is compromised, e.g. if someone publishes what an identifier represents, then the identifier can be used to look up the public information. An analogy is the abuse of social security numbers in the USA. With only your social security number, which is supposed to be only an opaque identifier for private data, banks credit bureaus can collect and sell to this data because it is widely shared among trading partners. This type of policy failure is however outside the scope of a technical standard.
Privacy and HIPAA conformance
The proposed framework allows manual and automated management and processing of competency and assessment data without exposing individually identifiable information, as required by HIPAA. Everywhere in the model, individuals and their competencies or learning assignments are represented by opaque identifiers. Where necessary to secure privacy, the identifiers can be obfuscated by common agreement among trading partners. For example, the "person or group" identifier in a competency record might be an opaque handle that can only be resolved to a true identifier through a resolution system that requires proper credentials.
Personalized assessment plans
Because a valid assessment is expensive—in time for all involved, not counting possible licensing fees for the use of the assessment instruments—it makes sense to try to limit the assessment to specific competencies. It also makes sense not to include competencies for which there is already credible data, if the existing data is more credible than what could be gathered through the assessment. Personalized assessment plans take existing competency data into account to help define what to test and how to test it.
In the proposed framework, the results of personalized assessments can be distilled into standard competency records. The assessments can produce very rich results that can be archived for auditing and analysis, but in that case only a distilled version of the results makes it into the data used for operational control of the rest of the framework, like adaptive learning. This helps keep the model simple. If richer behavior is needed, built-in audit trails allow more custom functionality without invalidating the basic model.
Figure 24 - From personalized assessment plans to competentency records
Several parameters govern how an assessment will be conducted:
|
Proficiency level |
In some cases, the purpose of the assessment is just to ascertain whether a competency is mastered. In that case, no proficiency level needs to be specified. The result is true or false. If you’re about to jump from an airplane, you don’t want to know whether your parachute folder does it perfectly 75% of the time. For example, either someone knows how fold a parachute properly, or doesn’t. In other cases, the assessment may be prescribed to verify a threshold measure of satisfaction. For example, you may want to check whether someone can throw a football accurately at a range of distances at least 75% of the time. A proficiency level might not be specified at all when you request the assessment, but you want it as a result of the assessment. |
|
Competency model |
An assessment is typically takes place in the context of some competency model. The competency model may be implicit, explicit, or just assumed to exist but poorly defined. The framework defines a standard way to specify an explicit competency model that includes a reference to the tested competency. The model might also be used to drill down into component competencies, or facets of competency for the tested competency. An assessment for a competency often actually tests component competencies, or facets of competency. For example, a math exam may test knowledge of the applicable theories, as well as the ability to perform specific operations to demonstrate a variety of specific skills and aptitudes. The framework defines a standard way to specify explicitly the competency model that includes the component competencies or facets to be assessed. |
|
Validity and reliability |
The higher the stakes, the more important it is that the assessment be valid and reliable. There is usually a trade-off.—the more valid and reliable, the more expensive an assessment instrument will tend to be. |
|
Security |
The higher the stakes, the higher the security requirements. |
Note that context was not cited among these parameters. Many assessments are designed to be administered in a particular context. For example, defensive driving skills may have to be evaluated in an actual road test situation. However, there does not seem to be any way to represent something as complex and variable as a context using a standard data model. Therefore, this kind of contextual information should be considered to be specific and intrinsic to the assessment itself.
Using a competency model to specify what to assess
In many cases, an assessment is ordered for competencies that may be defined at a fairly high level of abstraction. The corresponding RCD might be something like “Can communicate effectively in writing.” Operationally, the assessment should contain items to test finer grained competencies or facets.
If a competency model exists that defines the component competencies and/or facets for the RCD, that model can be used as a guide to assemble the necessary test items, based on the RCDs referenced by the model. Different organizations or communities of practice may have different ideas as to what this means, and so they may have different competency models associated with this RCD. The competency modeling of “Can communicate effectively in writing” is likely to be quite different for a book publisher and for an auto repair shop. By specifying a competency model as well as the RCD to be assessed, it is possible to get the assessment you need based on the needs of your organization or community of practice.
Since the standard competency evidence record includes a way to trace back the source of evidence, if you get a competency record that says that Joe is proficient in “Can communicate effectively in writing”, you can look up the evidence and the source of evidence, and decide whether the competency model that was assumed and the form of assessment satisfy your requirement, without having to do a full analysis of the detailed competencies.
Formulating an assessment request
We can now revisit the assessment request model that was summarily described above. A generic assessment request can be formulated by specifying the following parameters:
|
Parameter |
Description |
Value space |
|
Competency definition |
Required. Identifier of a RCD that defines the competency to be assessed. |
Identifier of a RCD |
|
Proficiency level |
Optional. A particular proficiency level to be achieved. If no proficiency level is specified, it is assumed to be “100% proficient”. |
Real number in the range -1 to 1 – same range as in SCORM and IEEE score. |
|
Competency model |
Optional. A structured competency model that includes a reference to the RCD. |
Identifier of a reusable competency model |
|
Confidence |
Optional. A confidence rating on a standard scale. The factors contributing to confidence may also be specified in more detail as validity, reliability and/or security ratings. |
Real number in the range 0 to 1 |
|
Validity |
Optional. A specification of the expected validity for the assessment instrument to be used. |
A validity rating, and a reference to the source of the rating. |
|
Reliability |
Optional. A specification of the expected reliability for the assessment to be used |
A reliability rating, and a reference to the source of the rating. |
|
Security |
Optional. A specification of the minimum security level for the administration of the assessment. |
A security rating, and a reference to the source of the rating. |
Figure 25 - Assessment request data model
Getting assessment results
A generic assessment result for a specific competency can be provided by using the data model in Figure 26. Results may be collected into larger records in which some of the data is normalized.
|
Parameter |
Description |
Value space |
|
Competency definition |
Required. Identifier of the assessed competency. |
Identifier of a RCD |
|
Proficiency level |
Optional. The measured proficiency level that was measured. If no proficiency level is specified, it is assumed to be “100% proficient”. |
Real number in the range -1 to 1 – same range as in SCORM. |
|
Competency model |
Optional. Competency model used, if any. |
Identifier of a reusable competency model |
|
Confidence |
Optional. A confidence rating on a standard scale, as proposed by the source of the assessment result. The factors contributing to confidence may also be specified in more detail as validity, reliability and/or security ratings. |
Real number in the range 0 to 1 |
|
Validity |
Optional. If a validity rating is available for the assessment instrument, what the rating is. |
A validity rating with a reference to the source of the rating. |
|
Reliability |
Optional. If a reliability rating is available for the assessment instrument, what the rating is. |
A reliability rating with a reference to the source of the rating. |
|
Security |
Optional. If a security rating is available for the way the assessment was administered, what the rating is. |
A security rating with a reference to the source of the rating. |
|
Normative score |
Optional. If the assessment resulted in a normative score (where the proficiency is evaluated against the proficiency of a population of other test subjects), the value of the score. |
T-Score |
|
Instrument identifier |
Optional. Identifier of a reusable assessment instrument used for the assessment, e.g. identifier of a SCORM package, or ISBN of a test workbook. |
Identifier |
|
Instrument |
Optional. Metadata describing the instrument that was used for the assessment. |
Subset of LOM metadata |
|
Details |
Optional. “Drill down” data about each of the component competencies or competency facets that were assessed. |
Assessment result records; may be nested n levels deep. |
|
Digital signature |
Optional. A digital signature that can be used to authenticate this assessment result and verify that it has not been tampered with. |
Digital signature including public key of the signer, checksum, etc. |
Figure 26 - Assessment result data model
Figure 27 - Combining repositories, competency records and personalized assessments
Targeted assessments and quizzes
A targeted assessment is geared to assess specific competencies or competency facets for a particular person or group, rather than a competency domain. For example, a hiring manager might order an assessment of job-specific computer programming skills in order to identify most qualified candidates for a position.
A quiz is a form of targeted assessment, in which only a few specific, typically fine grained competencies are being tested.
If credible competency records exist for some of the competencies or facets to be assessed, and the evidence is satisfactory, the corresponding parts of a standard assessment may be omitted. The result is a targeted assessment. Omitting the unnecessary parts of what would otherwise be a one size fits all assessment can save time, money or both. At the extreme, if credible competency records exist for all the competencies to be assessed, a simple review of the competency records may be sufficient and it is not necessary to administer a formal assessment.
The framework does not specify whether targeted assessments must be used. But if they are desired, it provides the necessary plumbing to construct such assessment.
Adaptive assessments
An assessment may also be adaptive. Adaptive assessments change their behavior and structure depending on the subject’s responses and detected abilities. A typical adaptive assessment might be structured to do some gross testing at first, and then proceed with more specific tests. For example, a math ability test might determine early on that the subject does not understand algebra, or might get this information from available competency records before administering the test, and therefore can save the time and embarrassment of presenting test items about which the subject is clueless. Or, an adaptive assessment might detect strengths in particular facets of competency and change the testing method accordingly.
Test item bank automation
Digital test items, test sections and exams can be stored in digital repositories as SCORM packages and described with LOM metadata. If the metadata for the test items in a repository include a classification element that designates a RCD, you can have instant virtual test item banks. Additional LOM metadata element may describe additional features of the test items that can be useful in test item selection. As test items need to be updated or rotated for a particular competency, they can be added to the repository and a search by RCD identifier will automatically find them.
Once you know which competencies or facets you want to assess, the digital test items can be automatically referenced and re-aggregated into a SCORM package that can be used to administer the assessment. If multiple items are associated with the same RCD, SCORM 2004 sequencing rules can be used to automatically select a random subset of those items for delivery in the assignment. For example, if the RCD specifies the ability to resolve quadratic equations at a certain level of complexity, there might be multiple test items designed to test this ability, typically with different problem values.
By specifying a competency model for the RCD to assess, the test items for the component competencies and/or facets of competency can also be automatically included into the test. For example, there may not be valid test items for something as vague as “conjugate the present tense of French verbs” but there may be test items for “conjugate the present tense of regular French verbs” and “conjugate the present tense of irregular French verbs”. The finer the model granularity, the finer the granularity of the test items can be. Of course, validation of the proper set remains a task that only instructional designers, subject matter experts and test designers can do. Automation does not replace common sense.
Reusable assessments as SCORM 2004 packages
SCORM 2004, whether through sequencing or through a single SCO that references multiple objectives, is particularly well suited to do reusable assessments that can be used in targeted or adaptive assessment applications, because the objectives for activities and objective maps that map to the SCO tracking data use identifiers and status data that map exactly to the status information in evidence and competency records in this framework.
However, as it currently stands, the SCORM does not specify a way to connect to a framework like this one in a standard way. This is in part a chicken and egg problem, of course. Without a framework it is not possible to define a specification, and without the specification it is not possible to test the framework. The good news, however, is that adding such a connection can be very simple and does not break any existing functionality.
IMS QTI
QTI compliant test items can of course also be stored in a repository, described by standard metadata, and massaged for delivery. They might be run through a rendering engine to generate a self-contained SCORM package, or administered directly through a QTI compliant delivery environment. One big issue with QTI as of this writing are the complexity of the rendering when test items use every defined features. This is further complicated by the extensions many QTI adopters want to add to their tests and test items. However this is outside the scope of this framework. A more difficult issue is the absence of a standard summary result data model for the current version of QTI (version 2.0), which makes it impossible to integrate the testing results into operational decisions or data flows directly. Every QTI based item or test may need a custom distillation process to generate the simple data used in the framework.
Targeted, adaptive learning plans and packages
|
Don has now enough data to formulate a training plan. Based on the results of the assessments, learning resources will be specified and implemented or acquired for each of the competencies to be developed or maintained. The training plan must also be heavily customized for the employees, because the assessments showed a lot of differences in the existing proficiency, with some employees very knowledgeable and ready for most emergencies, some almost entirely clueless, and others somewhere in between. The instructional design team adds the learning resources to the company’s content resources repository as they are being developed. Non-digital resources like textbooks and classroom training are also represented by digital resource descriptors. Each learning resource is tagged by highlighting in the competency model the specific competency it will develop, so that it can be found easily when a learning assignment for that competency is created in the training plan. |
Figure 28 - Developing learning content |
Don doesn’t have the time, budget or resources to design a custom training plan for each employee. The unit managers cannot be expected to have enough expertise in office emergencies to design a plan for their employers. Fortunately, the company is using a learning management system that can automate much of the process. For each employee, the learning management system proposes a custom learning plan. Some managers choose to review the learning plan and tweak it, based on their scheduling needs and what they know about their people. Others are content to just let their people follow the automated learning plans, but still request reports showing the employee progress and success in meeting the competency requirements.
The original deadline set by upper management to get everybody on the same page in order to satisfy the insurance company and company policy is looming close. Don can monitor progress toward this goal as he reviews daily readiness reports generated automatically by the learning management system. When he finds that one unit is falling significantly behind, he has a brief talk with the unit manager. That is usually enough to jog the procrastinators into catching up.
Targeted learning plans and packages
The goal of targeted learning is to train or educate a particular person or group for specific competencies or competency facets, rather than providing education for a full competency domain. For example, a project might require that developers acquire some new job-specific computer programming skills, or an employee who is engaged in inappropriate behavior might be required to undergo some remedial training to address the cognitive, affective and behavioral aspects of the proper behavior. Skill gap analysis can determine which competencies need work.
Figure 29 - Skill gap analysis
If credible competency records exist for some of the competencies or facets to be developed, and the evidence is satisfactory, the corresponding parts of the learning plan may be omitted or made optional. The result is a targeted learning plan, which might be embodied in a SCORM package representing the learning activities and the desired outcomes. Omitting the unnecessary parts of what would otherwise be a one size fits all learning plan or package can save time, money or both. At the extreme, if credible competency records exist for all the competencies to be developed, a simple review of the competency records may be sufficient to indicate that it is not necessary to undergo the training. If nothing is required, the learning plan is in effect empty or just contains optional activities.
Credibility is of course all important here. In the example of the misbehaving employee, evidence of competency from a supervisor should always be considered more credible than evidence collected from assessments results gathered by an online training package.
The framework does not specify whether targeted learning plans or packages must be used rather than generic training packages. But if they are desired, it provides the necessary plumbing to construct such plans or packages.
In a subsequent section we will see how the training plan can be automatically or semi-automatically populated with learning resources to become a learning package.
Adaptive learning plans and packages
Learning plans or packages may also be adaptive. Adaptive learning can change learning activity sequence and choose different methods depending on the learner’s success and other factors. A typical adaptive learning plan might be structured to use some assessments or use existing competency record data to determine how to proceed with the actual learning. For example, a math learning package might determine early on that the subject does not understand algebra, or might get this information from available competency records before administering the test, and therefore would provide introductory tutorials until the learner is ready to move on to the more substantive topics. Or, an adaptive assessment might detect strengths in particular facets of competency and change the tutorial method accordingly.
SCORM 2004, whether through sequencing or through a single SCO that references multiple objectives, is particularly well suited to do adaptive learning, because the objectives for activities and objective maps that map to the SCO tracking data use identifiers and status data that map exactly to the status information in evidence and competency records in this framework.
However, as it currently stands, the SCORM does not specify a way to connect to a framework like this one in a standard way. This is in part a chicken and egg problem, of course. Without a framework it is not possible to define a specification, and without the specification it is not possible to test the framework.
Automated and semi-automated learning packages
Once a learning plan has been determined on the basis of the competencies to achieve, it is necessary to find learning resources to execute the plan.
Digital learning resources or digital representations of non-digital learning resources can be stored in digital repositories. They can be stored as SCORM packages and described with LOM metadata. If the metadata for the learning resources in a repository include a classification element that designates a RCD, you can have instant virtual learning resource banks. Additional LOM metadata element may describe additional features of the learning resources that can be useful in learning resource selection. As learning resources need to be updated or improved for a particular competency, they can be added to the repository and a search by RCD identifier will automatically find them.
Figure 30 – An integrated competency management, assessment and learning framework
Once you know which competencies or facets you want the learner to acquire, the digital learning resources can be automatically referenced and re-aggregated into a SCORM package that can be used to deliver the learning. If multiple resources are associated with the same RCD, the most appropriate resource can be selected automatically, semi-automatically or manually, depending on how the learning plan is managed. For example, an Army unit commander might use automation features to generate a basic learning plan and to identify possible learning resources, but then make manual selections of the resources. The definition of learning sequence may also be partially automated, using strategy templates to guide the construction of a learning sequence adapted to the type of competency to be developed. The templates may facilitate the inclusion of remediation, choice of learning method, and so on. (See Ostyn presentation at ADL Plugfest 9)
SCORM 2004 sequencing rules can be used to automatically select a random subset of those items for delivery in the assignment. For example, if the RCD specifies the ability to resolve quadratic equations at a certain level of complexity, there might be multiple test items designed to test this ability, typically with different problem values.
By specifying a competency model for the RCD to acquire, the learning activities for the component competencies and/or facets of competency can also be automatically included into the learning package. For example, there may not be learning resources for something as vague as “conjugate the present tense of French verbs” but there may be learning resources for “conjugate the present tense of regular French verbs” and “conjugate the present tense of irregular French verbs”. The finer the model granularity, the finer the granularity of the learning resources items can be. Of course, validation of the proper set remains a task that only instructional designers, subject matter experts and test designers can do. Automation does not replace common sense.
Dealing with unavoidable changes
Right in the middle of the deployment of the new adaptive training plans OSHA just promulgated new regulations that affect the office emergency readiness curriculum. Some of the reusable competency definitions in the competency model are now obsolete, and new ones must be added, along with the corresponding learning resources.
Chris doesn’t panic. She knows Don is ready for this. Don makes the necessary additions and changes to the company competency models and the instructional design team installs the new assessments and learning resources. As soon as this has been tested, Don has the learning management system send a memo to everyone affected explaining that there will be new stuff to learn, and to expect that some changes may be slipstreamed if they are already engaged in the training, and the changes are published.
Using the framework, changes to competency requirements or to learning resources can be made at any time, and any automated process can automatically adjust. This requires no artificial intelligence.
Obviously, though, published SCORM 2004 packages that are not dynamically assembled do not get updated. This is not necessarily a disadvantage, since in many cases, especially if complex sequencing logic is involved; there is an implicit “contract” with the learner that the learner will complete or succeed in a well-defined learning task. Having the task changing constantly may be disconcerting and counter productive. However, in some cases, it is important that the learning be as up to date as possible. For example, in a financial institution you don’t want to train financial advisors on products that have been withdrawn, while missing out on the product introduced last week. On the other hand, bank policies and customer relation skills training may be learned using static packages that need to undergo careful legal and management review as a whole package before they are published.
The proposed framework supports both scenarios: Static learning packages, and dynamic learning packages.
Adaptive performance support
It is now clear to Don that some of the competencies required for office emergencies will not be retained without ongoing training and repeated drills. Also, upper management has made it clear that “just in case” training for every possible emergency is out of the question because of the high cost in resources and time this would involve. There must be a way to help ensure adequate performance even when someone has not yet been trained, or if the benefits of the training have decayed beyond usefulness. This should take the form of job aids that are judiciously placed where then can be seen in an emergency, as well as the removal of some barriers to performance in an emergency.
The proposed framework cannot address some critical performance issues, which are beyond the scope of what can be remedied through training and job aids. Issues like process improvement to remove the obstacles to performance, motivation and a positive working atmosphere are beyond what can be fixed by the application of technology, no matter how judicious it is.
However, even when those issues cannot be addressed due to enterprise or organizational inertia, the framework can help improve performance until they are resolved
Adaptive job aids
Using the framework, if a system is aware of a person’s existing competencies, the system can automatically prepare relevant job aids as a person engages into a task. For example, if a call center employee is asked to handle a new product line, the job aids can be automatically selected based on the employee’s prior experience with this or similar product lines.
In a military setting, existing competencies in a soldier’s record may be used to determine what job aids must be made available for a mission.
Figure 31 - Performance support
Just in time training
Using the framework, if a system has access a person’s competency records, the system can automatically escalate from job aids to finely targeted tutorials. For example, if a call center employee runs into a new problem that requires handing off to a supervisor more than a couple of times, a specific tutorial finely tuned to avoid wasting time can be provided to the employee to use during idle time.
In a military setting, existing competency records in a unit’s record may be used to determine what additional training the unit requires for a planned mission. Since different learning resources may be targeted to the same competency, and there is a standard way to find those resources, the most appropriate resource for the situation can be selected.
Data for analytics to support process changes
Ultimately, job aids and training are often required mostly because of defects in the business processes or operational workflows themselves. By tracking the use of job aids and remedial, just in time training, the framework provides statistics that can be used indicate where improving the processes or workflows will have the most impact, or to diagnose sick processes.
Putting it all together
Figure 32 – Putting it all together: Simplified framework exploiting standard competency data
The black arrows and the grey zones in Figure 32 represent a general flow of data and processes that inform the competency modeling and the development of assessment and training resources. Those flows and processes are not amenable to standardization, because every enterprise or agency has its own culture, priorities and processes that govern them. The colored arrows represent interfaces between different services that implement the assessments, training and competency data management. These services may be part of the same system, or implemented as separate, cooperating systems in a Service Oriented Architecture (SOA). The arrows also represent "choke points" through which data travels from one service to another. The choke points allow a straightforward enforcement of security, privacy, data filtering and data processing policies (e.g. confidential topics, HIPAA conformance, etc.)
The same framework also support readiness assessments based on competency records, personal portfolios, and Just In Time job aids and other performance support resources tailored to the requirements of any task.
The lightly patterned gray arrows with grey caption represent shortcuts. One kind of shortcut shows how tracking data from learning activities can be used as immediate feedback to influence an automated adaptive learning plan. For example, in SCORM 2004 sequencing, the status of objectives can influence the adaptive sequence. Another kind of shortcut is the possible use of personal portfolios or portable personal profiles to store competency records. For example, an ePortfolio for a person might contain a personal learning plan (desired learning outcomes and a specification of the activities to each those outcomes), work products and competency records.
This will be a technical section describing the basic data models assumed by the framework
For the RCD data model, see the IEEE Standard Draft
Other data models (most already in existing presentation)
Strawman Competency Map node
Strawman Competency Tree
Strawman Reusable Competency Map
Strawman Competency Evidence Record, Competency Record
Strawman Specified Competency Reference (HR-XML)
Strawman Competency Evidence collection (normalized models)
(to be done)
This will be a technical section describing the core services assumed by the framework
(to be done)
This will be a technical section describing how CORDRA can be exploited by the framework.
(in progress)
CORDRA (see http://cordra.net)
If RCD identifiers are handles, as defined in the Handle system (see http://handle.net ) or the handle-based CORDRA specification, the handle itself can be used to find a reference to the current version of the RCD.
The Handle system specified the use of a globally unique opaque identifier that, beyond its use as a globally unique key, can also be used to invoke a resolution service to look up specific named values. This is similar to the way a DNS record can be drilled into to locate specialized entries through a DNS look up.
Using CORDRA, it should be possible to look up, for example:
· Given a RCD (Reusable Competency Definition) identifier that is a handle,
o The actual original RCD record (i.e. URL to retrieve an XML document)
o If one exists, the identifier (handle) of the most current known version. This handle can be used for forward chaining to the current version.
o If one exists, the identifier (handle) of a canonical competency map for the decomposition of the competency into facets and/or components.
o If one exists, the identifier (handle) of a canonical competency map that provides context for the competency, i.e. a map in which the competency is a facet and/or a component.
o The current owner of the rights to the RCD, if different from what might be specified in embedded metadata in the RCD
o If one exists, the URL of a version of the RCD that includes translation into a specific language. Additional translations may be added after the original RCD has been published. Since an RCD should not be modified once published, this provides a mechanism to add translations while still using the original identifier and leaving the original intact.
· Given a RCM (Reusable Competency Map) identifier that is a handle,
o The actual original RCM record (i.e. URL to retrieve an XML document)
o If one exists, the identifier (handle) of the most current known version. This handle can be used for forward chaining to the current version.
o If one exists, the identifier (handle) of the main RCD that defines the competency detailed in the competency map.
o If one exists, the identifier (handle) of a canonical competency map that provides context for the RCM, i.e. a larger map in which the RCM represents a facet and/or a component.
o The current owner of the rights to the RCM, if different from what might be specified in embedded metadata in the RCM
o If one exists, the URL of a version of the RCM that includes translation into a specific language. Additional translations may be added after the original RCM has been published. Since an RCM should not be modified once published, this provides a mechanism to add translations while still using the original identifier and leaving the original intact.
How to do this with CORDRA is not entirely defined yet, but is likely to be defined in the months to come.
(to be done)