Connecting Experimental Psychology and Psychometrics

I combine the two methodological approaches of experimental psychology and psychometrics to study people's anshin-kan (sense of peace of mind) and working professionals' "state of being" (not well-being, but being). In this article, instead of discussing the specific research projects that I am currently conducting, I will address research methods in psychology.

Two broad methodological approaches are commonly used in psychological research. One is an experimental approach that measures behavior and physiological responses, such as brain waves and heart rate. The other is an approach that uses questionnaires to measure psychological states. Questionnaires are often criticized for issues such as dishonest responses, response bias, and the difficulty of discerning the true intentions of respondents. In particular, researchers who measure physiology or behavior sometimes argue that questionnaires are too subjective. I began my research career in experimental psychology and later incorporated questionnaires, so I am acutely aware of the strengths and weaknesses of both approaches. From my perspective, I can say that questionnaires and behavioral or physiological data are not opposing methods; rather, they are complementary. In this article, I will adopt a somewhat sympathetic stance toward psychometrics, as it is often misunderstood and frequently criticized within research practice.

Behavioral and physiological data are often regarded as objective and accurate. Nevertheless, such data alone cannot fully account for a person's psychological state. For example, even if a physiological response such as a fever is observed, one cannot determine whether its cause is a cold or stress. Similarly, even when individuals exhibit the same behavior, the psychological processes leading to that behavior may differ depending on the person. In other words, whether the psychological construct one intends to measure is being measured appropriately is an entirely separate issue.

Questionnaires face major issues such as respondents who answer carelessly and the inclusion of inaccurate responses. However, behavioral experiments are subject to comparable forms of misconduct, including failure to follow instructions and intentionally invalid responses. Even so, improper behavior in experimental tasks rarely becomes a topic of discussion. In research using questionnaires, obtaining reliable data requires rigorous evaluation of quantitative indicators such as reliability (whether the same results are obtained consistently) and validity (whether the instrument accurately measures the intended psychological construct) (Nunnally & Bernstein, 1994; DeVellis, 2017). The same forms of evaluation are also necessary in research using behavioral or physiological measures. Thus, in psychological research, we must recognize that every method has its limitations and we need to address these limitations appropriately.

Even when a questionnaire demonstrates high reliability and validity, it is difficult to entirely eliminate the major problem of inaccurate or invalid responses. However, there are several strategies to mitigate their impact. I will introduce six representative approaches below.

1. Avoid difficult-to-answer questions
Questions that use overly complex language, are too technical, or address socially sensitive topics (e.g., attitudes related to discrimination) tend to yield unreliable responses and are therefore poorly suited for questionnaires. In such cases, it may be more appropriate to employ behavioral observation or physiological measurements.

2. Randomize the order of questions
The order in which questions are asked should be randomized to prevent the order of questions from influencing responses.

3. Enlarge the sample size
Collecting responses from many participants helps reduce the relative impact of invalid responses. However, as the sample becomes larger, even very small differences can appear statistically significant. For that reason, it is important to also check effect sizes and other related measures.

4. Introduce a two-stage survey
Surveys can be divided into a screening survey (a preliminary survey to narrow down the target participants) and a main survey. Identical questions (for example, asking the frequency of AI use) are included in both surveys to check the consistency of responses. Participants are also excluded if their answers contain logical contradictions, such as indicating in the screening survey that they use generative AI but then claiming in the main survey that they are unaware of it. Additionally, the screening survey evaluates open-ended responses (for example, asking the reasons for using AI). Participants are excluded if their answers contain an excessive number of typos, are irrelevant to the question, or are unintelligible. Furthermore, an appropriate response-time range should be established based on prior research, and in the main survey, participants whose completion times are excessively short or long are also considered to have provided invalid responses and are excluded. It is not unusual that more than 10% of participants are removed from the analysis through implementing these procedures.

5. Set multiple conditions and assign participants randomly
As in psychological experiments, it is essential to establish two or more conditions and randomly assign participants to each condition in order to control the bias arising from individual differences. For example, when comparing the brand images of the two casual restaurant chains in Japan, Saizeriya and Gusto, participants who use these restaurants evenly can be randomly assigned to evaluate one of the two brands. Using this approach, the effects of individual differences and any invalid responses can be evenly distributed across conditions. This makes it more likely that observed differences between the two conditions can be interpreted as meaningful both statistically and psychologically.

6. Verify reproducibility
Although it requires considerable time and resources, conducting replication studies enhances the reliability of the results.

Other precautions are also necessary; for example, avoiding the use of a convenient sample. It is only natural to view questionnaires as unreliable without these basic measures.

Finally, I would like to introduce a perspective that has greatly influenced my thinking. Cronbach (1957), who served as president of the American Psychological Association, stated that "research results in psychology often vary depending on the method used, and that these differences are valuable because they can lead to new discoveries." In psychological research, methodological or positional differences should not lead to conflict; rather, they should complement one another in the process of advancement. From not only a Cronbach's philosophical but also a methodological standpoint, Podsakoff et al. (2012) have emphasized the importance of combining multiple measurement methods and data sources, noting that reliance on a single method can give rise to common method bias.

Psychological research encompasses a variety of methods, including behavioral data, physiological measures, and questionnaire surveys. None of these methods is perfect, but combining multiple approaches allows for a deeper understanding of psychological phenomena. I conduct integrative research using both experimental psychology and psychometrics, focusing on familiar topics such as "peace of mind" and the "state of being" of working individuals. Research cannot be fully accomplished through a single method; rather, new insights emerge by integrating different perspectives. As noted by Dr. Cronbach, differences in methods and fields should not be seen as sources of conflict but as opportunities for new discoveries. Moving forward, I aim to contribute to the creation of a society in which people can maintain a sense of peace of mind by continuously integrating theory and empirical research. I also look forward to collaborating across disciplines and gaining new insights together with all of my peers.

---

Reference Literature

・Cronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12 (11), pp. 671 to 684. 
https://doi.org/10.1037/h0043943
・DeVellis, R. F. (2017). Scale development: Theory and applications (4^th ed.). Sage.
・Kyutoku, Y. (2021). Toward "Anshin-kan" Measurement using Questionnaire. Journal of Japan Society of Kansei Engineering, 19 (4), pp. 175 to 178. Japan Society of Kansei Engineering.
・Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3^rd ed.). McGraw-Hill.
・Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, pp. 539 to 569. 
https://doi.org/10.1146/annurev-psych-120710-100452