The Theory Informing DoView Planning
The new visual planning for organizations, initiatives, strategy, research and AI agents.
Quick Overview
DoView Planning is an applied planning approach arising from the theoretical work of strategy psychologist and outcomes theorist Dr Paul Duignan. He developed outcomes theory and its applied version, DoView Planning, after working on outcomes issues with hundreds of organizations across multiple sectors and settings. Outcomes theory consists of a set of definitions, frameworks, principles, and theoretical and practical tools for working with outcomes systems of any type. Two novel concepts at the heart of outcomes theory are, first, that any attempt to take action in the world, whether by humans or AI agents, can be conceptualized as an outcomes system. The second is that claims about a ‘This-Then’ logic, which underlie all outcomes systems, are best represented as a particular type of diagram called a DoView strategy/outcomes diagram.
With the growth of AI, Microsoft and others are now arguing that business planning should focus on outcomes, rather than sticking to traditional, more functional, siloed individual organization-based planning approaches that have been the norm up until now. Outcomes theory provides an ideal basis for a new outcomes-oriented approach to planning and implementing action in a world where humans and AI agents are increasingly collaborating.
This page provides information about some key outcomes theory concepts. If you want a very quick introduction to outcomes theory, watch the three-minute video below. If you want in-depth information about it, read Duignan’s seminal 2009 paper describing outcomes theory in detail in the context of evaluation and performance management. The rest of this page offers additional details on outcomes theory and more theoretical aspects involved in constructing and using DoView diagrams within DoView Planning.
What is covered on this page
Introduction
DoView Planning is an applied version of strategy psychologist and outcomes theorist Dr Paul Duignan’s work on outcomes theory. Duignan developed outcomes theory after consulting on outcomes issues for hundreds of organizations, including governments and international bodies such as the IMF. His work has encompassed strategy, planning, outcomes, measurement, performance management, outcomes-based contracting, commissioning, impact evaluation, alignment, organizational development and related outcomes-related issues in multiple settings. In addition to his applied strategy consultancy, he developed his theoretical approach during his strategy psychology doctoral studies focused on strategic evaluation; working in, and managing university and other research groups; being the director of a national-level Parliament research unit and serving as a Fulbright Senior Scholar at the Urban Institute in Washington D.C.
Duignan developed outcomes theory and its applied version, DoView Planning, in response to the somewhat confused and inconsistent approaches to outcomes issues he saw in many organizations across multiple countries. In contrast to the related field of accounting, which has a well-established body of theory, outcomes work does not currently have a single integrated theory. This situation has arisen because outcomes work involves professionals from a range of disciplines and backgrounds operating in a variety of settings.
With the rise of AI, Microsoft and others are now advocating that all planning should focus on outcomes rather than relying solely on traditional, siloed, functional approaches to planning and implementation. We now exist in a realm where multiple agents—both human and AI—are acting in the world. The most effective way to plan in such a world is to use a planning and implementation approach that is structured around outcomes thinking. It needs to be a new approach, such as outcomes theory and DoView Planning, which can handle planning and implementation in situations where implementation is carried out by multiple organizational forms, both long-standing and ad hoc working groups involving both human and AI agents.
2. The elements of outcomes theory
Outcomes theory is made up of a number of different elements. These include definitions, principles, frameworks and tools. The key elements in outcomes theory include the following.
Definitions of terms. Outcomes theory provides ways of defining terms used in outcomes work, for instance, it provides a precise way of describing and robustly differentiating the common terms used when working with outcomes systems. These are terms such as outcomes, indicators, outputs, inputs, accountabilities, controllable indicators, not-necessarily controllable indicators etc.
The concept of an outcomes system. This is the idea that underpinning any initiative or endeavor that is attempting to take action in the world is an implicit or explicit outcomes system, and that thinking in this way is the most useful way of looking at outcomes work of any type.
The DoView Planning Framework. This sets out the essential components that any outcomes system should include. It enables you to assess whether or not any particular outcomes system is well-formed. It can be used as the basis for delegation and contracting, and to clearly specify what an organization or initiative will report back on regarding its performance and success
The theoretical concept of a DoView strategy/outcomes diagram (or outcomes hierarchy). This is used as a conceptual tool within outcomes theory and provides conceptual clarity when discussing various aspects of outcomes systems.
The practical tool of DoView strategy/outcomes diagrams. These are used extensively within outcomes theory as practical tools when working with outcomes systems in its applied version DoView Planning.
DoView Drawing Rules. These set out how to draw a fit-for-purpose DoView suitable for use during within outcomes theory and the DoView Planning approach.
A typology of the types of elements that can be put in boxes within a DoView diagram. This typology solves the current terminological confusion that surrounds the definition of many of the terms used in outcomes work. For instance, what is an outcome, a result, and impact, a goal, an objective, a deliverable, an intermediate outcome, a long-term outcome, an accountability, a KPI etc.
DoView Planning. A comprehensive, integrated, applied outcomes theory-based approach for planning, contracting, implementing, measuring and reporting on any organization or initiative of any type.
Evaluation types framework. A framework for thinking about different types of evaluation.
Impact evaluation design types. A framework for identifying different impact evaluation design types.
Economic evaluation design types. A framework for identifying different economic evaluation types.
You can get more details on a number of these from Dr Paul Duignan’s early paper on outcomes theory and from the 100+ outcomes theory and DoView Planning tools in the DoView Handbook. The sections below set out some aspects of outcomes theory that are particularly relevant to constructing and using DoViews within DoView Planning.
3. Rules for Drawing DoView Diagrams
The DoView Drawing Rules are set out below. If you follow these rules when building a DoView, this ensures that it will be fit for purpose for use within each planning, implementation and measurement stage of any initiative.
Keep boxes within the DoView diagrams as short as possible for quick reading.
Name boxes as ‘outcomes’ – in the sense of completed states rather than as ongoing ‘processes’. Change the wording of a box from ‘…ing’ to ‘…ed’. For example, change ‘increasing stakeholder support’ to ‘increased stakeholder support’.
Only include a single concept in each box – e.g. don’t put ‘achieve A by doing B’ because such a box includes both an end and a means in the same box. For example, do not have a box reading ‘increased awareness through producing a video’; break this box up into two separate boxes. Doing this encourages consideration of alternative ways of achieving the higher-level box, which is ‘increased awareness’.
Build externally-focused, not organizationally-focused, DoView diagrams. Your thinking should go beyond just the things that an initiative is doing and things that are controllable by the initiative. The DoView should show all of the things that need to happen to achieve high-level boxes. This should include risks (written in the positive) and assumptions included within the same model. This means that a reader can see everything that needs to happen within the same DoView rather than having to look elsewhere for things like assumptions (for instance, in a separate table within planning documentation).
You do not need to divide a DoView diagram up into sections such as inputs, activities, outputs and outcomes. Allow the causal flow in the outside world to determine the position of the boxes within its structure. After you have built a DoView, if you want, you can then tag boxes within it with any of the above categories that you want to.
Do not siloize DoViews by requiring that a lower-level box can only contribute to a single higher-level box.
Keep measurement separate from the boxes that are being measured by mapping measures (indicators) onto boxes in the DoView after the diagram has been built. This means you can represent currently unmeasured boxes in your DoView. It also raises awareness that a measure is a measure of a box rather than necessarily being the box itself. This approach highlights that the way a box is being measured may be able to be improved in the future.
Future-proof the DoView by building one that shows everything that needs to happen over time. Then identify current priorities onto the DoView. This is much more efficient and transparent than just creating a DoView that only contains current priorities. It allows the selection of current priorities to be critiqued by just looking at the other boxes in the DoView that have not been prioritized at the particular point in time. It also makes strategic planning faster because you can rapidly change the priority of particular boxes rather than having to rewrite your strategic plan each time priorities change.
Do not assume that an organization needs a single high-level outcome. If it is working on multiple issues that are differentiated in the outside-world, it is appropriate for it to have multiple high-level outcomes.
Put outcomes into a left-to-right format with the highest-level outcomes on the right. This is a discretionary rule, and you can build bottom-to-top outcomes models, but for languages that work from left to right, a left-to-right model is generally easier for them to process.
Include all relevant and necessary steps at each level within the model.
If appropriate, put an evaluative descriptor such as adequate or sufficient in front of your steps and outcomes.
Make your DoView Diagrams as large as they need to be to capture the organization or initiative by dividing them up into meaningful sub-pages, creating a multi-layered model.
4. What exactly is an ‘outcome’?
When you are doing outcomes work of any sort, problems often arise due to a lack of clarity about what exactly is meant by particular terms. A great deal of confusion, time wasting and talking past each other can occur in the course of doing outcomes work. One reason this arises is when the same term is used for things that differ, or different terms are used for the same thing. The types of terms that are used in outcomes work include: outputs, objectives, outcomes, goals, short-term, intermediate, long-term outcomes, results, impacts, collective impacts, participatory outcomes, collaborative outcomes, and results.
How Outcomes Theory Deals With The Terminology Problem in Outcomes Work
Outcomes theory deals with the terminology problem in outcomes work in the following way. It simplifies things by first seeing every outcomes system of any sort having either an implicit or explicitly drawn DoView strategy/outcomes diagram. This DoView diagram lays out the claims about a ‘This-Then’ logic that underlies every attempt to act in the world. Outcomes theory then introduces a list of ‘features’ that boxes within a DoView diagram can have. This list is used to define the specific type of box being discussed in any outcomes-related discussion. Among the list of possible features of boxes that can be included within a DoView diagram, several are controllability, measurability and position within the DoView. So then, regardless of what someone wants to call a box within the relevant DoView, you can tightly specify it in terms of its features, so a result for some people’s purposes could be defined as measurable and at the top of the DoView. This approach provides an efficient and technically robust way to clearly define the different terms used in outcomes work, which were listed above, such as outcomes, impacts, results etc.
Applying this approach to the use of the term outcome
The level of potential confusion in outcomes work is illustrated by just considering the definition of the term outcome itself. An outcome is a relative, rather than an absolute, term. Within outcomes theory any type of organization or initiative is defined as an attempt to change a part of what is viewed as just one long, complex, and ongoing outcomes hierarchy (which is what DoView diagrams capture). Whether or not something is viewed as an ‘outcome’ depends on how the larger outcomes hierarchy has been divided up into smaller chunks in order to frame up what an individual organization or initiative is trying to do. An organization or initiative’s DoView diagram is a smaller chunk of a larger outcomes hierarchy. An ‘outcome’ at the top of one organization or initiative’s DoView can be viewed as an ‘input’ at the bottom of another initiative’s DoView because it is modeling a higher-level chunk of the overall outcomes hierarchy. This situation causes significant confusion when people try to define the ‘outcomes' for a particular organization or initiative. Within outcomes theory, this is all clarified by simply visualising the underlying outcomes hierarchy and the position of specific organizational or initiative subsection DoViews within it.
In formal terms within outcomes theory, an organization or initiative’s outcome is defined as one of the highest-level boxes appearing within the relevant DoView. This definition is silent on other features of such boxes, for instance, their measurability or their controllability by the particular organization or initiative (and hence their automatic attributability to that initiative merely by having been measured as having occurred). Obviously, when doing outcomes work, people may want to specify additional features of the boxes they are interested in. For instance, they can require that ‘outcome’ boxes be both at the highest level within the relevant DoView, but that they also need to be measurable.
This approach to defining terms such as outcomes, impacts, results etc. enables a concise way for anyone to clearly define what they are talking about when doing outcomes work. So they can, for example, say, ‘When we say we want you to give us your X’s, we want boxes within your DoView that are as high-level as possible but which also need to be measurable and attributable to your organization or initiative. So, if a purchaser/funder clearly states what they want in this way, it does not matter what they call the type of box they require, because a supplier/provider will immediately understand what is needed, because the features of the boxes in question have been clearly specified. Anyone who has been involved in outcomes work for long enough will realise that terminological confusion is a major timewaster in outcomes work.
5. Advantages of presenting strategy in the form of a DoView
A key aspect of DoView Planning is using DoView diagrams at every stage of the planning, implementation and measurement process. From a conceptual point of view, DoView diagrams can be seen as surfacing the underlying explicit or implicit ‘This-Then’ claims that underpin any strategic thinking. At the current time, within planning and implementation documentation, these claims are presented in various formats, but often just as a text-based narrative. Text-based narrative is not the most efficient way of ‘This-Then’ claims in a planning and implementation context. The advantages of presenting these claims visualized within a DoView diagram are listed below.
1. Cognitive Load Reduction
The DoView approach visualizes outcomes and the steps leading to them in a continuous left-to-right flow, which fits with how humans within many cultures process sequences. As a result, it reduces the mental effort required to understand the model, making it easier for users to grasp the logic and flow of the strategy.
2. Psychological Momentum
The continuous flow within a DoView fosters a natural sense of progression, using the deep metaphor of going on a trip to help users intuitively understand how early steps contribute to final outcomes. This makes the strategy more engaging and easier to follow.
3. Consistent Mental Models
By maintaining a consistent logical structure for each part of a DoView, the approach helps users build a stable mental model of the strategy. This makes it easier to identify areas for improvement and understand the overall logic behind the initiative.
4. Natural Flow of Controllability
The DoView approach mirrors the natural progression from more controllable steps on the left to less controllable outcomes on the right. This helps users transition from concrete actions to more general outcomes, making it easier to understand the increasing complexity and the broader impact of each step.
5. Use as a shared thinking tool
Once in use, an initiative’s DoView becomes a shared thinking tool for decision-makers, stakeholders and those implementing an initiative. It helps to bring people’s mental models out of their heads, ensuring everyone works from a shared understanding of the underpinning logic of an initiative. The DoView then acts as a common authoritative model of what is being aimed for. It can be used for multiple planning and implementation purposes within DoView Planning. In addition, it can be used to communicate what is being planned and implemented across all levels of an organization or initiative through to those delegated or contracted to carry out the initiative on the ground. Attempting to plan an initiative without a DoView as the visualisation of what is being planned is like trying to build a high-rise building without any drawn plans, or hoping to plan a military expedition with no map of the terrain you want to fight in.
5. Synergy Between Humans and AI
Now that we are in the age of AI, an initiative’s DoView can be used by both humans and AI. The version that humans use will contain less detail than the version of the DoView that AI can handle. The AI version can capture all of the known relationships between boxes within the DoView. The human version is the same fundamental model, but it is abstracted to reduce the cognitive load for humans who are attempting to work with it. Humans get a clear and manageable overview, while AI can capture into deeper layers of relationships and data, leading to better decision-making. As people become increasingly relaxed about using DoViews and dependent on them for planning and implementation, new methods of leveraging AI’s more advanced analytical abilities will be able to be developed. Hopefully this will lead to the progressive introduction of richer and more dynamic DoView visualizations (see the Proposed Collaborative Opportunities Page).
6. DoViews as a visual language for decision-making
In any strategic decision-making situation there is a set of questions that need to be answered. These include the following.
What are the end points you are trying to achieve (high-level outcomes)?
What are the possible ways (lower-level steps) through which you might be able to achieve these outcomes?
What evidence is there for each of the proposed lower-level steps being effective at achieving the higher-level outcomes?
What are the assumptions you are making about what is necessary to achieve higher-level outcomes?
What are the risks that you need to avoid in order to achieve outcomes?
How are you attempting to measure what is being done?
How much of each lower-level step has already been achieved at any point in time?
What additional information do you need to decide on what to do next?
What priority do you want to give to focusing on each lower-level step at a particular point in time?
Are the projects/activities you are undertaking directly aligned with your priorities?
How are you going to control delegation or contracting to make sure that projects/activities are actually doing what they were specified to do when planning them?
How will you evaluate implementation and impact?
How do the specifics of what you are doing in your initiative fit into the larger overall strategy for an organization, sector, region or country?
High-level decision-making requires that decision-makers have as much information as possible regarding the answer to each of these questions as they make decisions about how to act. Currently, high-level decision-makers' discussions about the next steps they plan to take tend to be rather unstructured. For instance, there is often no certainty that different decision-makers around the table share the same mental model of exactly what is being attempted in an initiative they are deliberating on. This lack of structuring and disciplining high-level decision-making discussions is in contrast to the multiple methodologies and disciplines that are employed at lower levels to feed into decision makers’ deliberations (e.g. cost-benefit analysis, options analysis).
Within DoView Planning, the attempt is being made to put DoView outcomes/strategy diagrams at the heart of all strategic discussions at all organizational levels. DoViews are an attempt to introduce more rigor and discipline into strategic decision-making. This is with the aim of DoView diagrams ultimately providing a generic, comprehensive visual language for identifying and communicating the key pieces of information that decision-makers need to have in an immediately accessible form at the moment they make their decisions.
Because DoViews are a new tool being introduced into busy decision makers’ lives, it is important to proceed step by step to gradually get people used to using DoView diagrams. As a result, the simple way in which DoViews are currently set out (e.g. not showing the full ‘wiring diagram’ of possible connections between boxes) has been deliberately designed to keep the cognitive load low for time-poor, busy decision-makers. However, progressively, as decision-makers get used to using DoViews, it can be expected that the way DoViews are represented will become increasingly sophisticated to enable them to provide an increasing amount of information to decision-makers which is immediately available at the moment they are called upon to make strategic decisions. Advances in the use of AI in the representation and use of DoViews is likely to assist with this.
In any decision-making situation, the information about a particular initiative, which is listed below, is currently scattered across multiple documents, databases, presentations, simulations, and verbal discussions. Ultimately, DoView Planning aims to collect all this information and model it in an accessible format within a DoView diagram, making it immediately available to decision-makers as they discuss their plans. The current iteration of DoViews only include some of this information in order to keep the cognitive load of using them as low as possible while their use in planning is being introduced.
‘This-Then’ Claims
The basic causal claims as to how each box may affect any other box within a DoView diagram. For instance, ‘if you do X, then Y will follow’. There are a number of different types of relationships. (e.g., direct cause, necessary precondition, probabilistic dependency, feedback loop, etc.). At the moment within DoViews this is implied by the positioning of boxes from left to right without filling out the full wiring diagram because this would overload the diagram with too much information. However, such information can potentially be captured within a more detailed DoView underlying the one that is used in planning meetings. AI could potentially manage a parallel more detailed DoView to the one that human decision makers are using.Evidence for Each Link
All available data or reasoning to support (or question) the ‘This-Then’ claims made regarding the relationships between boxes. This can be further broken down into: the quality of the evidence, such that it is (e.g. high/medium/low confidence); how much evidence has been collected (studies, observations, reports, arguments from theory); the difficulty of collecting evidence in regarding to the particular ‘This-Then’ claim. It is important to have this information to differentiate where decision-making is based on evidence and where it is decision-making under uncertainty is taking place. At the moment, this information is not shown in the standard DoViews that are being used, but over time, ways of visualizing it without overwhelming the reader can be developed, and again this information could be managed by AI.
Current Extent of Achievement
It is important to know how much progress has been made on the boxes within an initiative’s DoView. In the standard DoViews being used at the moment, this is indicated by traffic-lighting boxes to show how much has been achieved with a particular box at the current point in time.
Priorities for taking action
A priority ranking for each box indicating how much focus there is going to be put on achieving it at the current time. Determining the priority of a box is determined by how far it is believed the box has already been achieved; how difficult it is to make progress on it now; and available resources to expend on it versus those needed for other competing priorities.
Assumptions
The assumptions as to how things need to be in order to achieve the outcomes being sought. In traditional planning, these are often listed in a table as a separate piece of documentation. However, within DoView Planning, assumptions are included as boxes within the DoView to make them visible so that decision-makers can directly interrogate the DoView in real-time as they are making their decisions, and have all the relevant information they need immediately available. Including assumptions within the DoView reduces the cognitive load of decision-makers having to remember a list of assumptions which is detailed in some other part of their strategic planning documentation.
Risk/Uncertainty as Positive-Framed Boxes
Potential pitfalls (e.g. “housing market doesn’t collapse” in strategic thinking leading up to the 2008 financial crash) are also included in DoView diagrams so that decision-makers continually factor them into their decision-making in the same way as they do with assumptions. Risks are worded in the positive and treated like any other box within a DoView that decision-makers need to take into consideration.
Alignment of proposed actions with priority higher-level boxes
Ensuring that projects/activities are tightly focused on priorities with a DoView is essential for the success of any initiative. Doing DoView Visual Alignment within a DoView ensures that alignment information is immediately available to decision-makers. When working with an initiative that involves many projects, a major flaw of traditional approaches to planning is the absence of a practical way of checking whether or not there is tight alignment of activities with priorities.
Stakeholder-Specific Perspectives
Different stakeholders’ views can be accommodated within DoView Planning. For instance, different stakeholders may have varying perspectives regarding how much particular boxes have been achieved at a particular point in time or what the priority should be for focusing on specific boxes going forward. This can be captured by having different versions of an initiative’s DoView marked up in accordance with the perspective of different stakeholder groups.
8. Abbreviations used in DoView Planning
Abbreviations that can be used in DoView Planning
M — Measure/indicator. M001 etc. (All measures/indicators are numbered sequentially, they are not separately numbered on the basis of who reports them).
M-Acc — Measure/indicator that is an accountability. M001-Acc etc.
M-Acc-Tar — Measure/indicator that is an accountability with a target. M001-Acc-Tar etc.
M-Acc-Tar-$ — Measure/indicator that is an accountability with a target and has a dollar value incentive or disincentive associated with achieving it. M001-Acc-Tar-$ etc.
M-Sup — Measure/indicator reported on by the supplier/provider. M001-Sup, M001-Acc-Sup, M001-Acc-Tar-Sup, M001-Acc-Tar-$- Sup etc.
M-Purch — Measure/indicator reported on by the purchaser/funder. M001-Pur, M001-Acc-Pur, M001-Acc-Tar-Pur, M001-Acc-Tar-$-Pur etc.
M-Oth — Measure/indicator reported on by another party. M001-Oth, M001-Acc-Oth, M001-Acc-Tar-Oth, M001-Acc-Tar-Oth-$ etc.
MProject — Measure/indicator project/collection. A way in which measures/indicators are collected (e.g. a particular survey that collects multiple measures/indicators). MProject-Sup, MProject-Pur, MProject-Oth etc.
EImplem — Implementation evaluation. EImplem-001 etc. (All evaluation questions are numbered sequentially; they are not separately numbered on the basis of whether implementation, process or impact evaluation). Elmplem-001. EImplem001-Sup, EImplem001-Pur, EImplem001-Oth etc.
EProc — Process evaluation EProc001, EProc001-Sup, EProc001-Pur, EProc001-Oth etc.
EImpact — Impact evaluation EImpact001, EImpact001-Sup, EImpact001-Pur, EImpact001-Oth etc..
EQues — Evaluation question. EQues001, EQues001-Sup, EQues001-Pur, EQues-Oth etc.
EProject — Evaluation project. EProject001, EProject001-Sup, EProject001-Pur, EProject001-Oth etc.
Outcomes theory offers a comprehensive framework for conceptualizing and working with outcomes systems of any type. Outcomes systems are systems attempting to identify outcomes and effect change. Outcomes theory encompasses a set of widely applicable definitions and principles that span the various terms employed by different disciplines involved in outcomes work. Its insights can be applied to human agents, AI agents, or both human and AI agents working together.
Difference between an outcome and an output
10. Set of 3-minute videos on aspects of outcomes theory relevant to DoView Planning
What is outcomes theory?
Discusses the differencebetweenanoutcome and an output. An output is controllable by a human or AI agent. This means that merely measuring it has occurred means that we can assume it occurring is attributable to the particular agent. This is not the case with higher-level outcomes where merely measuring outcomes often does not guarantee that changes in them can be attributed to a particular agent because such high-level outcomes are often influenced by a number of other factors. Always discussing outputs and outcomes against a DoView diagram makes discussions about the different level of outputs and outcomes much clearer.
Building better KPI and indicator sets
When developing indicator sets for human or AI agents within any type of outcomes system, it is essential to do so against a visual DoView outcomes model that outlines all the high-level outcomes being sought. If you don’t you have no way of knowing if your indicator set comprehensively measures the full set of outcomes and steps that you are trying to change in the outside world. To assess an indicator set, people have to implicitly build a mental outcomes model. Why not develop one in the form of a DoView and use it to build your indicator set from the beginning?
What are outcomes models (DoViews)?
Outcomes models, strategy diagrams or DoViews lie at the heart of outcomes theory. They are both a coneptual and practical tool. They set out in a visual format all of the high-level outcomes being sought by any outcomes system and all of the lower level steps that may be used to influence them. Outcomes models are drawn according to the DoView conventions, which means they are a generic way of modeling outcomes in any sector, and because of this, parts of them can be reused in different contexts.
Escaping the terminological madness in outcomes work
When dealing with outcomes systems (any system that attempts to specify outcomes and acts to change them), there are often a lot of different terms being thrown around which are interpreted in various, sometimes confusing, ways. Because it is based on a rigorous set of definitions, outcomes theory allows us to clearly specify from a conceptual point of view exactly what is meant by the use of a particular term such as outputs, outcome, goal, objective etc.
Identifying evaluation questions
In any outcomes system, assessing its impact is crucial. Just like with indicators, mapping evaluation questions to an outcomes system’s Doview outcomes model is vital. Evaluation questions exist at various levels within an outcomes model, and the same question can be phrased differently, leading to confusion. Determining the level where different evaluation questions lie is significantly easier using a visual outcomes model rather than attempting to do it mentally or discussing it solely in writing. The current discussions about the appropriateness of various AI benchmarks would benefit by mapping them onto an appropriate outcomes model.
Gaming indicators when used for incentivization
A concrete example of an outcomes theory principle. ‘The more an indicator is used for incentivization, the less accurate it will be. This applies to organizations but equally to AI systems. An AI’s ‘reward function’ is analogous to a performance indicator and just like some Wells Fargo employees AI systems can game their rewards which are being used to train them. So in AI systems as in organizational systems one needs to risk-manage around manipulability when a human or AI agent attempts to get a ‘reward’ in a way that is not aligned with the intentions of those who set the system up.
Assess versus assist mode in delegation and contracting
Difference between an assess mode and an assist mode in an outcomes system. If you have an outcomes system with a sole available provider focusing on one part of your DoView outcomes model, then you also need to have an assist mode for that provider. This is because, in comparison to situations where you could quit that provider and there are many others that could take its place, if there is only one available provider, you need to help them improve when they fail an assessment.
Different types of evaluation - implementation, impact etc.
Evaluation comes in various forms, and these can be positioned along the continuum of a DoView outcomes model. While evaluation often focuses solely on evaluating high-level outcomes, it is essential to recognise other types as well. These include implementation evaluation at an initiative's outset, process evaluation that looks at context and execution, impact/outcomes evaluation, summative evaluation, and economic evaluation. Given the current interest in evaluating AI systems, we need to consider employing all these evaluation types.