ProMES Meta-Analysis and Database
There is a good deal of research that has been done on ProMES and the amount is rapidly increasing. This has led us to ask the question of how best to accumulate, integrate, and disseminate this growing body of information. During the early years of the work on ProMES, it was easy to pass on the results of the ongoing projects to the few who were interested. That is not longer the case.
In dealing with this issue, it became clear that what we have in the international ProMES research work is a truly unique opportunity to document a body of research. We have a large number of field experiments done indifferent countries, by different researchers, in different kinds of organizations, with different types of organizational personnel, on different types oftasks. There are few, if any, comparable interventions where such data even exists, much less where it is carefully documented.
Thus, it became clear that the best way to deal with this growing body of knowledge about ProMES was to systematically gather it and put it ina form that ProMES researchers could use to compare the various results across different studies.
Such a database is a potentially powerful way to learn more about ProMES. To understand what circumstances lead to ProMES being more or less it seems to work, and how we can improve it, we need data from multiple studies. It is impossible for one or even a few researchers to do all the projects necessary to provide enough data to answer these questions. Thus, a formal, centralized way of collecting and aggregating project data was needed.
The specific mechanism we are using to aggregate and communicate the data is meta-analysis. For those unfamiliar with this methodology, the idea is fairly simple. You gather data on how the interventions did onthe important outcome variables. In ProMES this would certainly be the overall effectiveness scores that come from the system, but would alsoinclude data on changes in each measure, how long the system was kept in place, any attitude data that are available, etc.
You can then express the results of the interventions in a common metric. This is usually an effect size, which is the number of standard deviations of change from the baseline to the ProMES feedback. By using this common metric, you can compare how much effect the different studies had on the important outcome variables.
In addition, with meta-analysis you can also gather quantitative descriptions of a set of factors or variables that characterize ProMES projects. For example, you gather data on the type of organization, type of job, what initial attitudes toward the system were, how the process went, etc. With these project variables measured, you can also look at the relationships between the important outcome variables and the project variables. For example, two project variables are the degree of project support from management and the percentage of people in the work group who are in the design team.If degree of management support, as measured by a rating scale, is related to how large the effect of the system is on important criterion variables such as change in overall effectiveness, this indicates that degree of management support is important for achieving change in overall effectiveness.If variations in management support are not related to how much impact the project has on overall effectiveness, management support is not socritical.
Development of the meta-analysis Instrument
One of the most difficult steps in doing a meta-analysis is determining all the potential factors that could be related to the success of a ProMESproject. A great deal of work went into this effort. The resulting meta-analysis questionnaire is a listing of the project characteristics we thought might be important. This questionnaire is completed by the ProMES facilitator or the project leader.
To see the actual meta-analysis instrument go to this link
ProMES Data Base
Potential uses of the Data Base
We see many uses of the data base including:
» Determining how effective the ProMES intervention is on the outcome measures.
» Learning descriptive information aboutProMES such as the average amount of time it takes to do it, number of indicators developed in different settings, how the contingencies are shaped in different systems, etc.
» Comparing the degree of effectiveness by such factors as country, type of organization, type of unit personnel,etc.
» Determining what factors are the most important influences in making a ProMES project more or less successful.
» Determining how ProMES influences the perceptions and outcomes of individuals in ProMES units.
» Testing hypotheses about why ProMES works.
» Answering technical questions about ProMES.E.g., are there certain classes of contingency shapes? Are certain shapesusually found for certain types of measures such as for quality measures? Are improvements usually made on certain types of indicators before others?
» Doing research on the relationships between variables in the data base that do not directly involve ProMES. There would be data on a large number of important variables across many different types of work groups and organizations. Thus, it would be quite easy todo research where the unit of analysis was the group or organization. Itis normally very difficult to get good data for such research because itis difficult to get the same data on many different organizations. One example of this kind of research issue would be the relationship betweenunit member/management trust and the function of the unit (manufacturing,sales, service, etc.) Another example would be the relationship betweencentrality of the unit to the overall functioning of the organization and technological stability of the unit. A final example would be exploring the relationship between stress and job satisfaction in multiple settings or the relationship between stress and amount of feedback.
Collection of Meta Analysis Data
I see this data base effort as critically important. We have an opportunity to not only learn about an organizational intervention in a way that hasnever been done on this scale. In addition, we have an opportunity to create a data base that has many different applications outside of ProMES. Thus,I urge all those who are doing ProMES projects to help us in this effort.If you are doing a ProMES project, please complete the structural database instrument and when it is ready, the individual perceptions instrumentas well.
To get the instruments, you can simply print them from this web site.If that is a problem, please e-mail the webmaster or Bob Pritchard directly to get a copy.
Access to the ProMES Data Base
Access to the ProMES data base means receiving a copy of the complete data base. The data base will grow over time and updates will be available from time to time. Who should have access to the ProMES data base turns out to be a somewhat complex issue. While the specifics are still being worked out, the following policies are definite.
» The purpose of the ProMES data base isto increase our knowledge of ProMES and the other variables in the database. All efforts to increase such knowledge are to be encouraged. However,there are some limitations to the use of the data base that are required to protect the interests of those contributing.
» Only those who have made substantial contributions to the data base will have access to it. Substantial is not yet clearly defined, but my expectation is that it will require submitting at least three completed projects (i.e. with baseline and feedback data)to the data base. To have access to continuous updates of the data basethe researcher must continue to contribute to it over time.
» No one can take the data from another person's project and report it as an individual study. Any such reporting of individual projects is the right of the researchers doing that project.Only reports of data aggregated across multiple projects can be reported.
» Bob Pritchard retains the right to publishsummaries of the overall ProMES related data in English.
» The data base cannot be passed on toothers not contributing. The only exception is when someone who has access to the ProMES data base is collaborating with another researcher and will be an author on any publication or presentation that come from the data.In that circumstance, the collaborators must agree to the other conditions listed here.