Framework Tests Methodology

Introduction

The strategy for testing GRI’s frameworks for evaluating the impact of using assessment techniques on performance is quasi-experimental, using case studies. This approach requires organizational environments where variables and performance criteria can be tracked over time. This is the approach used for testing during Step 1 when building the first framework. The same approach was used for the general framework of steps 2 and 3.

Phase 1 Case Studies

During phase 1 of the project, two case studies were analyzed in parallel to closely follow the process by which leaders' use of the assessment technique improved the organization's performance. It was possible to contrast the collected data of the two cases. A single case limits generalizations and makes it difficult to prevent certain biases, such as incorrectly judging the representativeness of a single event, exaggerating the salience of data due to its availability, or biasing estimates through unconscious anchoring.

Several cases increase external validity and help prevent against observational bias. Yin argues that the logic behind the multiple case approach is similar to that guiding multiple experimentation and that each case should be selected so that it (a) predicts similar outcomes (a literal replication), or ( b) produces opposite results but for predictable reasons (a theoretical replication). In the first two case studies, the first literal replication strategy was selected.

Phase 2 and 3 Case Studies

In the second phase of the project, the test cases were extended to companies, techniques, and leaders in new industries of varying sizes and across various countries, although the coding protocol and interviews were less formal than for the first two case studies. During phase 3, the framework was extended, and the availability of GRI’s adaptive profile and tools enabled faster analysis and testing.

Construct Validity

For the construct validity test to be satisfied, care must be taken to select the specific events to be studied (in relation to the general and specific questions of the study). One must also be able to demonstrate that the measures selected for these events truly reflect the specific types of events selected.

Contamination phenomena can occur between secondary and primary data, and the subsequent transcription of certain primary data during note-taking is naturally affected by two biases: forgetting and a posteriori rationalization.

Three tactics are generally used to increase construct validity. The first involves using multiple sources that support converging lines of inquiry. The second tactic is to establish a chain of evidence. The third tactic is to have the case study report essay reviewed by key informants. In phase 1’s exploration step, the first two tactics were employed. A systematic transcription of observations and a rigorous coding approach to both primary and secondary sources helped limit biases. In the test step, the third tactic was applied.

Internal validity

Internal validity concerns only studies of causes and explanatory studies. When the first framework was tested in step 1, constructing explanations and analyzing events over time in the two case studies best supported this form of validity. Testing the framework on new case studies in step 2, while expanding it to new applications, uses, and users in step 3 for the same cases, further strengthened the internal validity of the framework's evolution into its new version.

External validity

External validity concerns whether findings can be applied beyond case studies. Statistical generalization, used in surveys and laboratory experiments, should be distinguished from analytical generalization derived from case studies.

A theory must be verified through replication of findings. Once replicated, results can be applied to a wider range of cases. Because the exploratory analysis of the uses of assessment techniques and their impact on performance was conducted across people, techniques, and organizations of varying quality, it helped build stronger theoretical frameworks, preventing constructions that are incompatible across cases and enabling generalization.

Reliability

The purpose of the reliability test is to ensure that if other analysts follow exactly the same procedure as described by a previous analyst and conduct the same case studies repeatedly, the later analysts should arrive at the same findings and conclusions. The role of reliability is to reduce errors and biases in the study. Therefore, the procedures followed were documented thoroughly. All necessary operational steps were carried out. Documents and interviews were organized to facilitate the audit.

Privacy

When using assessment techniques, since the transcripts and analyses mention the names of people and companies in operation, they are, on the one hand, coded so that these names do not appear, and, on the other hand, communicated only in a confidential manner.

A coding procedure is used for the names of people whose profiles are published, and a table is created to establish the correspondences. Given the large number of companies and people, pairs of first and last names, as well as company names, were changed: this allows translations to be made in both directions and to reverse translations for analyses if necessary. The analyses of translated personal names are considerably slowing down the analyses, so the original names are kept as long as possible, and translations are only made during the final coding phase.

Measurement Tools

For individual analyses, the adaptive profiles measured by GRI provide a concise and detailed description of people’s social behaviors that is also adaptive and can evidence engagement and performance.

For macro analyses, the measures from the adaptive profiles also provide a synthetic understanding of an organization's functioning and its performance. They enable qualitative and quantitative analyses of groups of people. Tools are available to measure expectations in social behavior: how people expect others to behave, at a job and team levels, also referred to as PBI (Position Behavior Indicator) and TBI (Team Behavior Indicator).

Originally, some tools were developed specifically with MS Excel to analyze groups of adaptive profiles from the two case studies. Correlations and other statistics, which were initially done with SPSS and spreadsheets, gradually moved online. Starting in step 3, the GRI platform could automate calculations and support both qualitative and quantitative analyses. Distribution and evolution diagrams over time, along with means, standard deviations, and other statistics, could be calculated instantly. Comparisons between individual social behaviors and the adaptive profiles could be measured and assessed against expectations in jobs and teams. Other instant calculations include the following:

• Distribution of the adaptive profiles within groups
• Engagement levels in a group
• Adaptations in a group
• Min-Max and average profiles in a group.
• Distinct analysis of managers and their team members
• Indices de performance: indices 'iopsoc', 'iopstra' et 'iopolig' that measure strategic, social, and economic performance at a group level.