"Science versus the Stars:"
A double blind test of the validity of the NEO five-factor inventory and computer generated astrological natal charts
Alyssa Wyman and Stuart Vyse, Connecticut College
Journal of General Psychology: 2008, 134(3), pp.287-300
Abstract
"The authors asked 52 college students (38 women, 14 men, M age = 19.3 years, SD = 1.3 years) to identify their personality summaries by using a computer-generated astrological natal chart when presented with 1 true summary and 1 bogus one. Similarly, the authors asked participants to identify their true personality profile from real and bogus summaries that the authors derived from the NEO Five-Factor Inventory (NEO-FFI; P. T. Costa Jr. & R. R. McCrae, 1985). Participants identified their real NEO-FFI profiles at a greater-than-chance level but were unable to identify their real astrological summaries. The authors observed a P. T. Barnum effect in the accuracy ratings of both psychological and astrological measures but did not find differences between the odd-numbered (i.e., favorable) signs and the even-numbered (i.e., unfavorable) signs."
This is an interesting paper that attempts to be a 'new and improved' 21st century low-budget version of the famous Carlson double-blind test of Astrology (1985) published in Nature. The Psychological Test, the NEO-FFI was more successful than the CPI (California Personality Index) used by Carlson, in that subjects were (as is to be expected for this type of self created report) able to recognize the analysis of their own questionnaire answers. It was smaller than Carlson with 52 participants, but unlike its predecessor it lacks the involvement of consultant astrologers in both an advisory capacity and as participators and frankly, it shows. In addition, it replicates some of the flaws of Carlson and is fatally based on several faulty assumptions.
Note: This is a cursory critical review and please contact me if you consider that a significant point has been misrepresented.
Flaws
1. Insufficient experience for self-selection: One of the biggest disadvantages of the experiment is that the mean age of the subjects is 19.3 years. This group could be expected to have a low level of self-knowledge or high suggestibility being at an age where family, peers etc have a strong positive and sometimes negative influence. These students are unlikely to have been through the life-defining experiences such as marriage, kids, home ownership or jobs. Tests of self-selection involving potential need subjects who have at least been through their Saturn return (i.e. after the age of 30). On tests and re-tests on the California Psychological Index the median results after 25 years is .58 suggesting a high potential for re-evaluation from age 19.
2. Self-selection is easy when results parrot the questions. Self-selection is more successful when the analysis closely reflects the self-image or desired self-image of the individual at the time. Self-selection failed with the California Personality Inventory Carlson (1985) and the Minnnesota Multiphasic Personality Inventory (Sundberg 1955), when subjects were unable to recognise their psychological profile in a blind test to a significant level. Subsequent tests including this one show that with the right type of psychological analysis, a subject can identify him or herself. Really it boils down to how closely the analysis repeats the subject's response. So to take an extreme example, if a subject's answer gives a high positive answer to Are you confident at parties or in a public situation, the subject will expect that their outgoing nature should score highly. The whole experiment rests on the assumption that because participants could recognize their self-reported NEO-FFI profiles, self-selection is a valid measure for all types of personality analysis including astrology. However, as this is a classic logical fallacy, no meaningful conclusions can be expected relating to astrology.
3. Subjects desire for an idealised self-image. In these types of tests, no matter how honest people try to be, they will create and favour an idealised image of themselves which may not correspond to reality. So the response and the preferred report will correspond with the ideal. However, the question is what is the value of a test that merely reflects the students' self-opinion no matter how accurate? Astrology is different and arguably more useful in say career choice, as it identifies potential, which may be unrealised. The authors refer to a related test by Taylor and Brown (1988) suggesting that many people hold an unrealistically enhanced view of themselves.
4. A Valid Astrological Chart Interpretation? The astrological charts had limitations being the result of a computer program that lacks synthesis, (a powerful enhancement that identifies themes) which should occur in a human interpretation. Also, by cutting out what some astrologers consider a key factor in the chart: the aspects, the emphasis of the chart will change. A Sagittarian Sun Sign with the sun conjunct Neptune will show very different characteristics to a typical Sagittarian. Most astrologers will state that a chart must be judged as a whole rather than in part. Some astrologers consider the aspects to be the most important factor in a chart.
5. The Astrology Profile was more complex as it had 29 separate personality statements compared with the NEO-FFI which contained only 5 dimensions. This as the authors concede may have made the task of assessing the astrological profile more difficult than the NEO-FFI
When analysing personality, self-attribution (psychological tests) is subjective and astrological analysis is objective (in the sense of coming from an independent third party). The irony is that the researchers are rating the objective report by setting against the subjective as the standard measure of accuracy. It's like checking the accuracy of a device such as a thermometer or scales by canvassing people's opinions rather than using standard weights and measures.
Faulty Assumptions
The paper also contains a number of faulty assumptions which an astrologer might have corrected:
1. Can psychology do what astrology attempts to do? The first sentence claims that astrologers' natal charts and psychologists' personality profiles share a common purpose - to provide a description of the respondent's personality. This is misleading. (See the footnote for a full explanation of this point) Astrology (as generally practiced) attempts to identify the potential of an individual which for a 19 year old is mostly in the future, unrealised and largely unknown at this stage.
2. Stable (ie constant) Characteristics? Astrologers and personality psychologists assume that people possess stable characteristics. This assumption is not correct. People do have some core characteristics that remain stable (in the sense of constant) but people also change and grow. Someone who is reserved aged 19 might grow into a very confident adult having achieved success. So this second assumption is in my view incomplete.
3. Personality determined at conception or at birth? While the scientific evidence suggests that the genetic profile is determined soon after conception, there is no hard evidence that personality is entirely determined at this stage. The question of how much the genotype determines the phenotype is controversial in the field of psychology just as the nature v nurture debate is controversial in social sciences. While modern astrology places great emphasis on the astronomical positions at birth, most astrologers would consider that character is part of a formative process and that environment can also shape the outward expression of personality. Conclusions based on a narrow determinist view of personality formation could be faulty.
4. Science versus the Stars?
a. The title of the test assumes that astrology is by definition the opposite of science. It suggests that it is not possible for anything astrological to have a scientific basis. This is incorrect even at the simplest level such as tidal phenomena. As this experiment is performed by scientists under scientific conditions, any success of astrology suggests a loss for science - when in fact it would be a great victory for science, but perhaps not for fundamentalist sceptics. This hints at an internal bias towards the null-hypothesis.
b. The use of the word stars is astronomically incorrect since the only star involved in this study is the Sun as the test is designed around the planets in signs and not constellations.
5. Confusion of Signs?
a. The concept of favourable and unfavourable signs is one to which most modern astrologers do not subscribe. Signs are listed as positive and negative (male and female) but translate this literally into favourable and unfavourable is to misunderstand modern astrology at the most basic level.
b. Researcher, Ken McRitchie pointed out that in their list of these signs, Aquarius was listed as both favourable and unfavourable. This simple error might have been eliminated if someone with expertise in astrology had been consulted in the process or had peer reviewed the article.
Interesting results:
The conclusions show no bias produced by what they considered the favourable description of certain signs (odd numbered over even numbered).
Did astrological knowledge affect the results?
There was no correlation of knowledge or belief in astrology with the overall ratings of the astrological summaries, which goes against the artefacts claimed in the Mayo-Eysenck experiment. In fact, those who knew their sun signs were less likely to correctly identify their real astrological profiles than those who did not. This could suggest a reverse artefact that students who know the character traits associated with their sun signs deliberately went against them due possibly to the sceptical approach of the experiment - suggested from the title.
A preference for the real astrology report, but not significant In spite of the flaws in the experiment in self-selection of astrology charts, of those that rated one of the two astrology profiles as the most accurate overall, 55% chose the real report compared with 44.5% chose the bogus report. This is interesting but not significant due to the small sample size.
What criteria should be used for comparison of the two analyses? Psychological profiles report on known personality, while astrology reports on known personality plus potential that is often unknown to the individual. Comparing the astrological reports with the psychological personality feedback report for accuracy is like comparing a 'selfie' that has been edited with a personal photograph album. The self-portrait photo is a one-time snap shot and has meaning for that moment. An album of photographs of a person in various situations conveys the history over time.
Conclusion
The report is clear. It is supported by a few good references and much detailed statistical analysis. The conclusions are presented in a reasonable manner within the parameters of the experiment. However, compared to the Carlson Experiment or the Gauquelin studies, this appears to be no more than a low-budget classroom exercise by a student supervised by her professor. As a result of limited resources, the subjects were unsuitable, insufficient (N=52), the astrological charts were incomplete, not synthesised and of a poor quality (when compared to a live consultant) but sufficiently complex to challenge the inherent and well-known problems with self-selection. The counsel or involvement of an astrologer (beyond advice on the purchase of astrological software) might have overcome these problems or allowed for a more balanced analysis. The fundamental problems with this experiment including the elementary blunders listed should have been picked up at the peer-review stage. However, it appears that the editor of the Journal of General Psychology failed to realize that astrology is a complex subject with many operational variables. In any experiment some astrological expertise should be present including at the review stage.
Though the experiment was based on a number of faulty assumptions about astrology, the results served to confirm the expectations of the supervising Professor, Stuart Vyse. However, in this instance he did not give astrology as practiced by astrologers a fair chance of refuting his assumptions.
References
Brody, N (1994) Continuity and change in personal dispositions. American Psychological Association pp.59-81
Carlson, S. (1985) A double-blind test of astrology. Nature, 318, pp.419-425
Sundberg, N.D. (1955) The Acceptability of 'fake' versus 'bona fide' personality test interpretations. Journal of Abnormal and Social Psychology, 50, pp.145-147
Taylor S.E. & Brown, J.D. (1988) Illusion and well-being: a social psychological perspective on mental health. Psychological Bulletin, 103, pp.193-210
Footnote: Psychological v Astrological Reporting
While it is true that natal astrology involves identifying personality, this is not the same as identifying personality using a self-reporting psychological profile. Psychological profiles report on known personality, while astrology reports on known personality plus potential that is often unknown to the individual. Some psychological profiles will identify potential and some astrologers will do no more than describe personality in a way that the client ends up thinking - why did I bother to have that done since I know it already? But the two types of reports have different objectives even though most people, including some astrologers, lump both under the term of personality description.
One of the significant differences is that self-identified personality changes over time. According to CPP - publishers of the California Psychological Inventory "Test-retest correlations for high school students over a one-year interval range from .52 to .73 with a median of .66. Test-retest correlations for adults over a 10-year interval range from .49 to .85 with a median of .77."
"The reliability of the CPI has been assessed as to its internal consistency, as judged by ... test-retest (medians: 1 year=. 68, 5 year=.56, 25 year .58)." (Info Refuge 2010) To amplify this point about the evolution of the personality, test-retest results of the CPI after 25 years shows a median of .58. Or to put it another way after 25 years, half the subjects had an altered perception of their self image by a factor of 42% according to the CPI measuring system. Since self-knowledge tends to improve with age, we can only assume that this adjustment is to greater reflect their true selves. (Currey 2011)
So any self-reported psychological test of anyone under the age of 30 will show a considerable divergence with the astrological report for the same individual. So the assertion that astrologers' natal charts and psychologists' personality profiles share a common purpose - to provide a description of the respondent's personality is misleading. And though this does not make these types of tests impossible, subjects should be made aware that one report is addressing the past, present and projecting into the future and the other is only describing the present based on the current self-image. Even then the subject is not necessarily in a position to make a fair judgement in comparing the two different reports.
Info Refuge (2010), California Personality Inventory, retrieved July 2010
Currey (2011) U-turn in Carlson's Astrology Test Correlation Vol.21(2) p.12
The experiment was jointly conducted by Professor Stuart Vyse, who is well known in sceptical circles as a speaker at CSICON 2011 - (a conference organised by CSI formerly known as CSICOP and "dedicated to science and reason") a platform shared with sceptics, James Randi and Phil Plait. He has written a book on the Psychology of Superstition.
Robert Currey
 Share
|
 |
 Main Astrology News Page. Information, stories, theories and facts.
Index to past articles Over 50 articles relating to astrology.
McRitchie, K. (2014). Confirmation bias in the Wyman and Vyse experiment. Correlation 29(2) 26-40.
A published peer reviewed paper on this experiment by independent researcher, Ken McRitchie.
|
