15. Artefactual conclusions in psychological research.

The scope of generalizations depends on the research method that has been implemented.

Within a given study, conclusions may prove to be reliable (valid) or artefactual (unreliable, invalid).

Errors in generalization occur because the researcher violates the rules of logic (in relating general and particular premises) and substitutes the norms of hypothetico-deductive reasoning with the acceptance of unfounded arguments. As a result, the conclusions turn out to be artefactual.

Two main paths to artefactual conclusions can be distinguished:

1. Conducting an invalid experiment and, as a consequence, making erroneous decisions about the action of variables.

2. Insufficient control over the inference, i.e. violation of the rules of hypothetico-deductive reasoning and the making of incorrect inferences or incorrect generalizations.

That is, artefactual conclusions can be drawn even after obtaining reliable data in a valid experiment. There are two directions: unreliable conclusions in statistical decisions; and the transition to generalizations when unfounded arguments are used or when the logic of relating empirical results and theoretical hypotheses is replaced by subjective preferences in the direction of justification.

Rules for organizing reliable conclusions:

- implementing a method involves not only a substantive evaluation of hypotheses, but also compliance with the method-specific norms for relating results to the conclusions that can be based on them.

Directions of errors in conclusions that are made during generalization:

1) the inclusion in conclusions of statements that are unjustifiably taken as their grounds but are not empirically confirmed;

2) making illegitimate generalizations when transferring a dependency to other psychological realities;

3) substituting the norms of the experimental method with subjectively self-evident but logically unfounded inferences;

4) violation of the rules for relating theoretical hypotheses and empirical results.

Evaluating the reliability of conclusions:

1) Evaluating the correctness of decisions about the experimental effect. The problem of statistical inferences.

- the reasoning carried out in transitions between different levels of tested hypotheses

- the reasoning in justifying the interpretive link between experimental and theoretical hypotheses.

2) Taking into account the scheme of conducting experiments – evaluating the representativeness of individual data or of the «averaged» dependencies obtained for samples. (by chance or by regularity?)

Internal and operational validity ensure the reliability of the conclusion about a dependency within the research situation.

3) Generalizations beyond the situation (external validity, population validity).

For an experiment with scientific aims, generalizing the type of causal dependency proceeds in 3 stages (if a laboratory experiment took place):

1. We draw a conclusion about the type of dependency;

2. A conclusion about the validity of the theoretical model used;

3. Moving from theory to reality, a conclusion about the degree of correspondence.

For experiments with practical aims, the logic of inference is sometimes simplified.

The logic of drawing a conclusion should include the following components:

- the hypothetico-deductive path of reasoning with asymmetry of inference;

- the construction of experimental schemes, within the framework of inductive inference about the result of the action of the experimental factor and the possibility of a causal explanation of the change in the dependent variable;

- a conclusion about the experimental hypothesis based on analysis of the obtained effect by relating the result to an assessment of validity;

- justification of the substantive grounds for generalizing the dependency beyond the experiment.

Artefactual conclusions may be of the following kinds:

1) an incorrect conclusion due to incorrect statistical decisions;

(one may overlook the need to search for a «third hypothesis»)

2) an incorrect conclusion about the action of the independent variable due to inattention to a rigorous assessment of the experiment's validity;

3) incorrect generalizations due to failure to account for essential additional variables or errors in understanding the relationship between a theoretical statement and the (empirical) experimental hypothesis;

4) replacing or distorting the norms of experimental inference with value judgments, appeals to authority, and so on:

- Missing links in causal explanations.

( Between explanation and generalization).

Example: analysis of the interrelations between the variables «aggressiveness» and «preference for watching television programs with aggressive content» based on the use of the cross-lagged correlation technique.

- Illegitimate identification of the principal cause.

In models described, in particular, by systems of structural equations, the same variables (e.g. social status or ethnicity when measuring IQ) may occupy positions at different levels – those of measured or latent variables.

- Substitution of one statement for another.

An author may uncritically assume that one situation they have recorded guarantees the content of another, and report only the latter.

(in questionnaires: «the subject acts this way often» or «the subject thinks they act this way often»?)

- The error of value judgments

- Illegitimate appeals to authority.

(What matters is not the scientist's credentials, but their argumentation)

- Appeals to fact.

As a sufficient argument, the assertion is offered that the evidence is an «established/generally accepted fact».