Technical Questions

I have kept track of some of the questions I have been asked and provided answers here. If you have technical questions that are not answered in the ProMES books and articles, send them to me and I will try to answer them.
- Bob

Q: What exactly is "effectiveness " in ProMes?


A: ProMES contingencies relate the amount of an indicator to how effective that level of the indictor is. It is an effectiveness measure in the sense that effectiveness is normally defined as output relative to a standard. The standard in ProMES is the miimum expected level of performance, i.e. the zero point on the contingency.

Effectiveness is defined as:

Q: Why are ProMes contingencies non-linear?


A: One of the key features in ProMES is the contingencies, especially the non-linearity of these contingencies. People have sometimes wondered why they are not linear.

The argument starts with the notion that an indicator can have a number of consequences that have value to the organization. If we think, for example, of an indicator measuring how much output is produced by a machine, one obvious consequence would be the objects produced. However, other consequences also occur such as wear on the machine, use of materials, and opportunities for doing preventative maintenance on the machine.

Different levels of output of the machine produce different levels of the consequences. These different levels of the consequences have different values to the organization. Producing 10 objects is usually better than producing 5, less wear on the machine is better than more wear, etc.

Thus, producing 10 objects has a given level of direct value to the organization based on what revenue or profit they can earn from the 10 objects. However, there is also a series of costs to making the 10 objects that the group has some control over such as wear on the machine. In most situations, wear at low levels of output is a probably a fairly linear function of amount produced, but wear at high levels of output is probably a positively accelerating curve. Thus, at high levels of output, wear (a negative consequence) increases at a faster and faster rate. Another consequence, preventative maintenance must be done to the machine and there is no problem doing such maintenance unless the machine is being used to near capacity. At near capacity, there is no time to do this maintenance. Not doing the maintenance on schedule has a high cost in terms of the long-term production from the machine.

Thus, there are three consequences in this example: objects produced, wear, and preventative maintenance. Variation in each produces variation in value to the organization. The value of objects produced may be totally linear where more units produce more value, wear (expressed as units produced) produces costs (negative consequences) which get proportionally greater with higher production, and opportunities for preventative maintenance (also expressed as units produced) is a flat line until very high levels of production, then it shows a sharp drop (increase in negative consequences) because there is a strong negative consequence of the maintenance is not done.

ProMES contingencies reflect the sum of all the consequences that follow from different levels of output on the indicator. The overall value to the organization (ProMES Effectiveness) of each level of the indicator is the sum of the values of the resulting consequences. In this example, the total value to the organization of 15 objects produced is the value of the 15 objects, the costs of wear on the machine for making 15 units, and the opportunities for preventative maintenance when 15 units are done on a regular basis. Different amounts of units produced produce different sums. A plot of the sums of these consequences would be the contingency. This example would produce a non-linear contingency which was fairly steep and linear at the lower levels of output but gets progressively less steep at higher and higher levels of output.

The basic idea is that when multiple consequences are present, the chances are great that non-linearities occur. I suspect that for most indicators, multiple consequences are the rule, which helps explain why most contingencies turn out to be non-linear.

Another thing to remember is that you can make some of the consequences a separate indicator if you wish. For example, in the case above you could directly measure whether preventative maintenance is done on schedule as a separate indicator. If you did this, the shape of the contingency for number of units produced should change. It would not show as much of a relative drop in effectiveness near the top levels of output because the negative consequence of not doing preventative maintenance is omitted from that contingency.

In such a situation where you had a separate contingency for maintenance, high levels of output that result in not doing preventative maintenance would get a higher positive effectiveness score, but there would be a negative effectiveness score in the feedback report from not having done the maintenance. The ProMES Overall Effectiveness Score would reflect both.

Q: Is there a difference between the shape of the contingency and the confidence one has in that shape? Put another way In doing contingencies one can think about the shape of the function and independently consider the confidence a person or group has in the accuracy of their perceptions about that shape. Is this an issue that must be considered in ProMES?


A: My answer to that comes directly from the Naylor, Pritchard, Ilgen (1980) theory from which ProMES is based. Our feeling when we did NPI theory was that there were several mechanisms that produced contingencies. One was someone simply telling you what the relationship was. A second was modeling where through observation of what happened to others, you formed an impression of the contingency. The third was what actually happened to you. These three mechanisms operate for all the types of contingencies, including the result-to-evaluation contingencies that are used in ProMES.

The first mechanism can be someone telling you the overall shape of the contingency (e.g. Someone telling you on a new job that you need to wear a tie but as long as your clothes are clean and you don't wear jeans, you don't need to dress any better than that). This is a statement of the relationship between how you dress and how you are evaluated. However, the second and third mechanisms as well as some instances of the first one do not deal with the shape of the entire relationship, but are specific events that pair one level of one variable with one level of the other. For example, suppose you wear jeans to work and your supervisor tells you "don't wear jeans". This shows that one level of dress (wearing jeans) is paired with one level of evaluation (negative). Only through a series of trials do you determine the overall relationship between level of dress and how you are evaluated. Another example is where you do a piece of work and someone tells you that is really good work. You may know that in the future that level of work is considered good, but do not have a sense of how other levels of work are evaluated. By a series of these pairings, you build up the perceptions of the contingencies. This pairing can be evaluations that are made on you or evaluations made on others that you are aware of.

Thus, one can think about a contingency as a bi-variate distribution of events, essentially a scatter plot in correlational terms. The non-linear function going through the points is the contingency.

However, this still leaves the issue of how much confidence one has in the contingency function. That is, is it different to have a situation where the contingency is high and the people are sure of it vs. a situation where they think it is high and they are not sure of it? In terms of the bi-variate distribution, this translates into how close the various points are to the best fitting non-linear function going through those points?

The partial answer I have is that you cannot have both steep contingencies and lots of error around each point. In correlational terms, lots of deviation around the regression line means that by definition the slope of that line is shallow. Thus any lack of confidence (what would be considered as error in correlational terms) serves to reduce the slope and make a weaker contingency.

The reason this answer is only a partial one is because it does not take care of the situation where the contingency is seen as low and there is little confidence that that judgment is correct. In other words, does it mean the same if I know it is not important (shallow slope) vs. the case where I simply don't know and thus do a flat slope?

In theory these are two different situations. However, in practice, I don't think it is much of an issue. If the design team does not know what the contingency should be and no one else does either (e.g. other group members or management), it should be either dropped or the organization should investigate whether it is important.

This brings up yet another issue. Specifically, does the above discussion imply that when people have limited control on an indicator, that indicator could never be of high importance, i.e. have a steep contingency? If people have limited control over the indicator, this is not directly a product-to-evaluation issue in NPI terms. Not having control is an act-to-product contingency issue. If I do not have control over how much of the indicator I can produce, it means that there is not a strong relationship between how much effort I put into the acts used to produce that result (product) and how much of it I actually produce. Low act-to-product contingencies will reduce motivation and should be increased as much as possible by using measures that people do have control over.

However, act-to-product contingencies and product-to-evaluation contingencies are largely independent in the sense that both can be high, both can be low, or one high and the other low. If I know there are big differences in how different levels of the indicator are evaluated and I know what these differences are, this is a steep ProMES contingency (NPI product-to-evaluation contingency). I may know this, but not have much control over how much of the indicator I can produce. This is a low act-to-product contingency. Thus, to finally answer the question posed in the previous paragraph, it is quite possible to have high importance on an indicator that people have only limited control over.

Q: What are the criteria for promes objectives and indicators?


A: (It is frequently a good idea to give this list to members of the design team)

Objectives should meet the following criteria:
 They should be stated in clear terms.
If exactly that objective was done, the organization would benefit.
The set of objectives must cover all important aspects of the work.
The objectives must be consistent with the objectives of the broader organization.
Higher management must be committed to each objective.
Keep the number of objectives manageable, normally three to five.

Indicators should meet the following criteria:
Indicators must be consistent with the objectives of the broader organization.
If the indicator was maximized, the organization would benefit.
Indicators must validly measure the objective.
All important aspects of each objective must be covered by the set of indicators.
Higher management must be committed to all the indicators.
Indicators must be largely under the control of unit personnel.
Indictors must be understandable and meaningful to unit personnel.
It must be possible to provide information on the indicator in a timely manner.
Accurate indicator data must be cost effective to collect.
The information provided by the indicator must neither be too general nor too specific.
It is important to keep the indicators to a manageable number, usually no more than 12.

Q: What are the important things to do before the design team starts its work in a promes project?


A: It is easy to forget some of the steps that need to be done before starting a ProMES project. This is a checklist of things that should ideally be done.

Project Approval
The project must be formally approved at the highest level possible.
All interested constituencies such as management personnel, other units, worker's organizations, etc. have been involved in deciding to do the project.
Benefits and costs have been clearly explained to all.

Assessment of Initial Attitudes
Facilitators have assessed:
The degree of trust between unit members and management.
Whether unit members and management agree on what the objectives are for that unit.
Whether all see productivity improvement as valuable.
Whether all see improvement as requiring considerable effort and time.
Whether all see participation/acceptance as essential.
Whether there are any planned changes in technology for the target unit.
Whether there are any planned changes in the organization of the unit.
Whether there are any planned changes in unit personnel.

Degree of Management Support
You should get the following commitments:
Management should agree to provide public support throughout project until a final evaluation can be made.
Management should agree to provide resources for development and implementation (time, access to data, etc.)
Management should agree that the measurement system developed through ProMES will become the new way that unit is evaluated by management. That is, ProMES should not simply be seen as an experiment that goes with the existing way the unit is evaluated, it should replace the existing evaluation system.
Management should formally deal with certain issues that will come up in the project. These issues include potential job loss if productivity improves, unit member compensation if productivity improves, whether ProMES performance will be tied to pay, and how will the ProMES project be expanded if successful.

Supervision and Unit Members
1. The process of ProMES must be carefully explained, including potential advantages and costs.
2. The reasons why they were chosen must be explained.
3. Ideally, they have volunteered to try ProMES.
4. If the design team is a subset of the entire unit, the issues of with how whole group will be involved in the process must be decided by the supervisors and unit members before the design team starts its work.