The twentieth century never happened

[Gnosital]

Something I wonder about in the history of scientific method, is the role of peer-review in publication. It seems to me, that even a hundred years ago in psychology there was so little in the way of peer reveiw that a a person's career could be made out of a few blathering books about "here's some stuff I thought up". I think this was the case to some extent in all the sciences emerging from the Renaissance (psychology got a very late start after physiology fucked around with philosophy and they had a love child). Until there were enough actual peers to do the reviewing and until there was a means by which the printed information could be disseminated widely, there was a crucial element of the scientific method that was missing. So before effective means of peer review evolved, we had a generally lower quality of scientific rigor.

Whaddya you guys think ?

[a different tim]

scathach wrote

Something I wonder about in the history of scientific method, is the role of peer-review in publication. It seems to me, that even a hundred years ago in psychology there was so little in the way of peer reveiw that a a person's career could be made out of a few blathering books about "here's some stuff I thought up". I think this was the case to some extent in all the sciences emerging from the Renaissance (psychology got a very late start after physiology fucked around with philosophy and they had a love child). Until there were enough actual peers to do the reviewing and until there was a means by which the printed information could be disseminated widely, there was a crucial element of the scientific method that was missing. So before effective means of peer review evolved, we had a generally lower quality of scientific rigor.

Whaddya you guys think ?

I think there's something in this, but it's not the whole picture. If you look at, for example, early copies of the philosophical Transactions of the Royal Society (which I think is the oldest scientific journal) there's all kinds of weird shit, including perpetual motion machines, alchemical schemes, and an experiment to try to communicate over long distances using the severed heads of dogs. On the other hand, astronomical theses, for example, were widely (if slowly) circulated even before the advent of printing. Mathematicians and astronomers would try to find flaws in, say, Kepler or Copernicus. I would argue that the genesis of scientific method lay in the Renaissance discovery that ancient authorities were not always right, and that a "fact" about the world, once discovered, might be refined or thrown out in the light of later work, and this is the beginning of peer review.

The formalisation of scientific method, on the other hand, came much later. I think it's true to say - in fact, I've argued above - that a lot of early scientists would not concieve of themselves as doing science in the way we think of science now.

Incidentally, there's a certain amount of historical controversy about this. On the one hand, we can look back and say "Newton was using sound methodology when he was working on optics, but was batshit crazy when he was working on alchemy". From the point of view of reviewing his work as science, this is good practice, but it is bad history, because Newton himself wouldn't have made this distinction or seen the two areas of work as separate in this way, so it doesn't give us any insight into the historical question of what Newton thought he was doing. It arises in the history of science in a way that it doesn't in other fields because science can make a credible claim to be genuinely progressive - in fact one of the things that science does is precisely to judge past insights in the light of present knowledge, which is the very thing that history tries to avoid, because historians seek insight into why people were doing things at the time they were doing them. So history of science, as a field, has a tension between these two ways of looking at the past. In this thread I've been leaning towards historical analysis, because I think we're looking at things in a more historical sort of way.

With especial reference to psychology I think it's a complex picture. Sure, they were pioneers with no peers to review but there are a lot of other factors. A lot of the early work was being done by people who were not trained scientists but were aspiring to invent a new science. Principles had to be discovered from scratch, and it wasn't clear how scientific methodology could apply. I've been reading the Ledoux book (thanks for that!) and it seems clear that early on people thought the only way to access the mind was via introspection. Freud, apparently, did some early work tracing neural connections but abandoned it because he couldn't see how this could connect to "how the mind works" which is what he was interested in. Hence the "made up" quality of a lot of the early stuff - it's not necessarily a product of lack of peer review (or not only that) but also a product of the investigative method they felt constrained to use, which in turn was limited by the technical tools that, as pioneers in a new field, they had at their disposal. Later on, of course, we go the other way with behaviourism, which is methodologically sound in that it measures observables - in fact the only observables they could measure at the time - but was forced to treat consciousness as a black box.

I personally think that scientific instruments have a huge effect on what can be fruitfully investigated, because this kind of technical capability determines what can be experimented on and hence what falls within the purview of scientific method. Galileo needed a telescope before he could show that Jupiter has moons and that therefore not everything orbits round the earth - before this Copernican astronomy was purely mathematical but, as Choobus would no doubt point out, with no connection to experiment - the string theory of its day, perhaps. The current cognitive approach, according to this picture, is as much a product of new measuring techniques starting with EEG and going on to MRI scans and the like, as well as new ideas about animal experimentation, as it is of any radical change in methodology - the difference between it and behaviourism is that scientists have learned to measure things that were previously thought unmeasurable.

[Sternwallow]

scathach wrote

Something I wonder about in the history of scientific method, is the role of peer-review in publication. It seems to me, that even a hundred years ago in psychology there was so little in the way of peer reveiw that a a person's career could be made out of a few blathering books about "here's some stuff I thought up". I think this was the case to some extent in all the sciences emerging from the Renaissance (psychology got a very late start after physiology fucked around with philosophy and they had a love child). Until there were enough actual peers to do the reviewing and until there was a means by which the printed information could be disseminated widely, there was a crucial element of the scientific method that was missing. So before effective means of peer review evolved, we had a generally lower quality of scientific rigor.

Whaddya you guys think ?

I think that you somehow missed the many, highly dramatic feuds between early scientists. Newton v Leibnitz and others come to mind. You are right about lack of peer, or any other, comment except, of course, for royal or church approval in the early days. Once scientists began meeting together, at the Royal Society, for one, the scientists clashed head-to-head both in print and in fisticuffs on the side lawn. The society began printing the products of the scientists and naturally included dissenting papers as well as papers praising and extending the ideas of the original one. Peer review was very effective at weeding out crack[pot ideas from the beginning and needed very little to formalize its process.

I think it was Francis Bacon who said "N+1 heads are better than N".

[Choobus]

Peer review as we know it didn't really start until after the 2nd world war, and is partly a result of increased government funding. (Read "The great Betrayel; fraud in science by Horace Judson for an interesting account). I don't think that the quality was necessarily lower before peer review. For one thing peer review has its own problems (politics being the most egregious, in my opinion) But 100 years ago (say) science was less specialized, and so it is concievable that the editors of a journal would have sufficient knowldge to review material themselves. This is simply not possible these days due to the sheer volume and diversity of work printed in a typical journal, even if it is relatively narrow in its scope. I have just had to fight with some dumbass referees for 8 months to get a paper published in the Physica Review. It was a very vexing waste of time, made more so by the fact that one of the refs was clearly an idiot.

[Gnosital]

a different tim wrote

I think there's something in this, but it's not the whole picture.

I don't think it's the whole picture either, but I do see peer review as being a crucial component. After all, it’s one of the most important criticisms of bullshit like ID, and why we don’t think much of ‘research’ that was conducted in Dr. Fred’s garage and published in Good Housekeeping or Guideposts.

a different tim wrote

If you look at, for example, early copies of the philosophical Transactions of the Royal Society (which I think is the oldest scientific journal) there's all kinds of weird shit, including perpetual motion machines, alchemical schemes, and an experiment to try to communicate over long distances using the severed heads of dogs. On the other hand, astronomical theses, for example, were widely (if slowly) circulated even before the advent of printing. Mathematicians and astronomers would try to find flaws in, say, Kepler or Copernicus.

I was thinking primarily of early psych perspectives, which include some weird shit as well, even among the major players. I think the older physical sciences had already reached a point in their maturation long before psych was around. Principles of physics and astronomy and math already had widespread applications among the ancients, so those sciences had already gone through a sort of practical trial and error phase of peer review. Of course, I'm generalizing out my ass here, I know there were formalizations of theories throughout the history of science and that there were also significant didactic controversies over which version was "right".

a different tim wrote

I would argue that the genesis of scientific method lay in the Renaissance discovery that ancient authorities were not always right, and that a "fact" about the world, once discovered, might be refined or thrown out in the light of later work, and this is the beginning of peer review.

And I would argue back that an even more remote genesis of scientific method occurs in the trial & error efforts of prehistoric humans to start a fire or make a killing tool. Significant refinements along the way would include the formalizing of empiricism and rationalism in philosophy, and then of course the overthrow of ancient Greek dogma during the Renaissance would constitute the beginnings of peer review.

a different tim wrote

The formalisation of scientific method, on the other hand, came much later. I think it's true to say - in fact, I've argued above - that a lot of early scientists would not concieve of themselves as doing science in the way we think of science now.

True dat. And it happened that psych was formalized even later than the other sciences, and I think because early psychologists had the advantage of modeling other sciences, the field more rapidly progressed into a science than it might have. Well, at least SOME schools of thought in psychology have, though not all. I suspect that before long, however, there will not be a perspective or subfield of psychology that will be worth crap if it isn’t soundly based in neurophysiology. There may be a great schism coming in psych, in that the softer areas, like humanism and psychodynamic, will become less mainstream and more parapsychology-like, while cognitive and behavioral will incorporate more neuroscience and will be more like extensions of psychobiology. You see, we psychobiologists have known for a long time that we’re going to take over psychology and obliterate the dumb parts.

BWA HAHA HA HAAAA!

a different tim wrote

With especial reference to psychology I think it's a complex picture. Sure, they were pioneers with no peers to review but there are a lot of other factors. A lot of the early work was being done by people who were not trained scientists but were aspiring to invent a new science.

True, in fact, James was trained in medicine, so right away THAT was a drawback! At least Wundt was one of Helmholtz’ students, so he started out ok, but that introspection thing just didn’t work out for him very well.

a different tim wrote

Principles had to be discovered from scratch, and it wasn't clear how scientific methodology could apply.

Again, true. I’ve no argument with that. The more minds that worked on the problem, however, the better things got.

a different tim wrote

I've been reading the Ledoux book (thanks for that!)

You’re very welcome, of course, but I must apologize, I realize now that I should have directed you first to “The Emotional Brain”. I’m reading it now; it was published before “Synaptic Self”, and provides more basic systems information than the SS, so it might have been a more logical sequence. My bad.

a different tim wrote

I personally think that scientific instruments have a huge effect on what can be fruitfully investigated, because this kind of technical capability determines what can be experimented on and hence what falls within the purview of scientific method. Galileo needed a telescope before he could show that Jupiter has moons and that therefore not everything orbits round the earth - before this Copernican astronomy was purely mathematical but, as Choobus would no doubt point out, with no connection to experiment - the string theory of its day, perhaps. The current cognitive approach, according to this picture, is as much a product of new measuring techniques starting with EEG and going on to MRI scans and the like, as well as new ideas about animal experimentation, as it is of any radical change in methodology - the difference between it and behaviourism is that scientists have learned to measure things that were previously thought unmeasurable.

You’re right, of course, technological advancements in brain study techniques are the entire reason my subdiscipline of psychology exists. I’m not trying to propose that peer review is the only or the most important component of the modern scientific method; it’s just that the role it plays in science seems to me to be taken for granted or ignored in historical accounts of the scientific method. I’ve only read a few books on history of psych, however, BR Hergenhahn, DA Dewsbury and LT Benjamin, for example. They’re good books, but focus more on philosophical roots and perspectives. Dewsbury puts a nice personal touch in, he likes to chronicle the fights and the cheating and the plagiarizing that went on along the way.

[Gnosital]

Sternwallow wrote

I think that you somehow missed the many, highly dramatic feuds between early scientists. Newton v Leibnitz and others come to mind.

Oh i know about some of those, Golgi didn't think much of that upstart Cajal, and took the opportunity of their shared acceptance of teh NObel Prize to dis Cajal's neuron doctrine (which ended up being right). And Wundt and James could hardly be civil to each other, although I don't think they ever got into fisticuffs at a meeting. I did not intend in my first post to propose that peer review was the sum total of the scientific method, nor did I intend to say that there was a general lack of peer review until the 20th century. I was just sayin' that there wasn't much peer review going on in psychology until then, and probably not so much in other sciences until the Renaissance.

Sternwallow wrote

I think it was Francis Bacon who said "N+1 heads are better than N".

That Bacon guy was pretty smart for not being a scientist!

[Gnosital]

Choobus wrote

Peer review as we know it didn't really start until after the 2nd world war, and is partly a result of increased government funding. (Read "The great Betrayel; fraud in science by Horace Judson for an interesting account). I don't think that the quality was necessarily lower before peer review. For one thing peer review has its own problems (politics being the most egregious, in my opinion) But 100 years ago (say) science was less specialized, and so it is concievable that the editors of a journal would have sufficient knowldge to review material themselves. This is simply not possible these days due to the sheer volume and diversity of work printed in a typical journal, even if it is relatively narrow in its scope. I have just had to fight with some dumbass referees for 8 months to get a paper published in the Physica Review. It was a very vexing waste of time, made more so by the fact that one of the refs was clearly an idiot.

I guess I've read a few really awful psychology papers from the early 1900s that gave me the impression that any assclown with any degree could publish anything he (not she, of course) shat out his arse back in the day. Maybe the phenomenon is more germaine to psych than to other fields. Lucky youse guys.

The modern problem in peer review seems to stem from your work being reviewed by those who are not your peers, innit?

[Choobus]

scathach wrote

The modern problem in peer review seems to stem from your work being reviewed by those who are not your peers, innit?

it's a double ended dildo. If it is a real peer they likely know you personally (which may or may not help) and may even steal your work. If not, the probably don't understand the work. Either way the potential for getting fucked is certainly not negligible.

[Gnosital]

Choobus wrote

it's a double ended dildo. If it is a real peer they likely know you personally (which may or may not help) and may even steal your work. If not, the probably don't understand the work. Either way the potential for getting fucked is certainly not negligible.

True, and your area is probably even more incestuous as it were, than my own.

There is a certain person I worked with when I was a post doc, who did shit work, constantly "massaged" data and mis-reported his results. He also stole my work and totally plagiarized one of my papers. So I filed a scientific misconduct complaint with the university about him. He basically weaseled out of it on a couple of technicalities, because he baffled the review board with statistical bullshit (his unique specialty), and none of them knew their asses from a hole in the ground on stats, and also because I think the university just wanted to make the whole thing go away.

However, the bastard is a reviewer on several journals in my field, it's apparently all he does any more, because he hasn't published squat. So if he happens to get any papers with my name on it, he slams it. He also takes it out on my husband and his collaborators' papers. We know enough people in the field who know who he is and what journals he's reviewing, so we're wise to him now, and have him recused from our papers if we know he's one of the reviewers.

There are definitely some problems with the peer review process. There should be some kind of screening process for reviewers. And they should at least be respected by a majority of researchers in their area to get the job. As it is, I think any assclown can kiss an editor's ass and end up with the job.

[Victus]

We get the weirdest replies from reviewers. We had a paper under review and we got a reply to the effect of "lol, maybe you should familiarize yourself with this work here *insert reference to other paper*).

I read the paper, and it had no significance to the topc at hand as far as I, nor anyone else in the lab, could discern.

[Gnosital]

"lol"??????

Maybe the reviewer gave your paper to an undergrad to write the review. The fucktard I worked for did that if he didn't want ot be bothered with doing the review. What sux most about it is that you have little to no recourse on a bad review like that, except to patiently and carefully point out the flaws in the person's *cough* reasoning, and hope the editor doesn't also have craniorectumitis.

It reminds me of how some people try to point out the flaws in logic of theists around here. And it's just about as futile.

[a different tim]

scathach wrote

You’re very welcome, of course, but I must apologize, I realize now that I should have directed you first to “The Emotional Brain”. I’m reading it now; it was published before “Synaptic Self”, and provides more basic systems information than the SS, so it might have been a more logical sequence. My bad.

No problem. In fact I'm enjoying it, it makes a lot of sense. He says in "SS" that he's updating some of the stuff. I don't feel like I can't understand it for not reading the earlier book or anything.

anyway, to carry on.....There have been some interesting articles in NS and SciAm about the effect of the net and online publishing of papers on peer review. There seem to be a couple of perspectives on it:
1) Some people are worried that with anyone being able to publish on line, peer review will efectively be diluted, affecting the overall quality of papers and science in general.
2) People who run and edit journals are concerned that, because people can get stuff online, their journals will fold/be unable to sell onough copies to pay decent referees/etc.
3) On the other hand, some people seem to think that it will be liberating because a lot of journals suffer from the problems that you and Choobus describe (i.e. your referee may well know you, be influenced by personal animosity or friendship etc, the alternative being a ref who simply doesn't know the field well enough) and teh internets provide a way to get stuff out there anyway - you can still get a cite if another scientist finds it valuable.

There is also the related problem of journal proliferation - it seems very difficult now for anyone to keep up with all the published research even in their own field. I mean how do you tell if someone publishes something crucial as a two page abstract in the North India Journal of whatever, in Punjabi, and it never gets translated?

Whichever, I suspect that the current model may be about to change.

[Gnosital]

a different tim wrote

....There have been some interesting articles in NS and SciAm about the effect of the net and online publishing of papers on peer review. There seem to be a couple of perspectives on it:
1) Some people are worried that with anyone being able to publish on line, peer review will efectively be diluted, affecting the overall quality of papers and science in general.
2) People who run and edit journals are concerned that, because people can get stuff online, their journals will fold/be unable to sell onough copies to pay decent referees/etc.
3) On the other hand, some people seem to think that it will be liberating because a lot of journals suffer from the problems that you and Choobus describe (i.e. your referee may well know you, be influenced by personal animosity or friendship etc, the alternative being a ref who simply doesn't know the field well enough) and teh internets provide a way to get stuff out there anyway - you can still get a cite if another scientist finds it valuable.

There is also the related problem of journal proliferation - it seems very difficult now for anyone to keep up with all the published research even in their own field. I mean how do you tell if someone publishes something crucial as a two page abstract in the North India Journal of whatever, in Punjabi, and it never gets translated?

Whichever, I suspect that the current model may be about to change.

I hope so! I think the current model is in desperate need of change! I've always thought journal impact factors were an elitist bunch of bullshit, and hopefully that will also change with a new model of online publishing. Whether or not a paper even gets considered for a high impact journal seems to depend on who you know or whose name is included on your paper, rather than the merit of the work. A new model could somewhat bring everyone down to the same level.

I don't understand the journals needing to make money anyhow! It's not like they pay the authors, they get the entire content for free, and the authors have to pay the publishers for reprints of their own articles! At least in neuroscience, the reviewers don’t get paid at all, they do it for the honor and the glory and the power. And as far as I know, most editors don't get much for their work, either. So the primary cost for the publisher comes from what, printing and clerical support? You would think that institutional subscriptions to even low impact journals are costly enough to adequately cover the ink and paper.

There are some completely online journals in neuroscience, and a few others that make the articles freely available after a few months. They still use the same old peer review process for accepting submissions, though, and I think there is some old-school type resistance to any major changes in that. I just don’t understand why.

As for diluting the quality of the research, I think that’s not likely. Suppose traditional peer review were replaced by online commentaries. I doubt a researcher would want to look like an idiot in front of the entire world, so there wouldn’t be a high incentive to publish bullshit. And an online peer review process would be far more effective because it would get more than 5 people involved, be more likely to reveal personal biases for what they are (no anonymous posters!!!), and help to improve the research more quickly and efficiently. Of course, someone could always jock your work, but they can do that now, too. I think it would be a lot easier to spot a copycat by the posting dates. A crooked reviewer in the traditional system, however, could read an unpublished paper, turn it down or delay the publication by making ridiculous objections, all the while getting his own stuff submitted elsewhere to scoop the other authors. That couldn’t happen if the peer review process was done after the fact.

You're right about the massive proliferation of journals, for sure. There are ridiculous numbers of them already, and there is a lot of crossover in terms of topic; for example, a study on the effects of late prenatal exposure to a substituted amphetamine on a dopaminergic second messenger system with relevance to later addictive tendencies would go into journals related to development, physiology, pharmacology, neuroscience, behavior, or any number of journals that combine those areas of research. You can only keep track of so many.

[Choobus]

I hate impact factors. IT's nothing more than a self fulfilling prophecy. It reminds me of bottled water. SOme journals (typically, the high imnpact factor ones) actually charge you for page submissions. Not only reprints, but you are encouraged to pay a "voluntary" fee (perpage) to "defray the costs of publication". It's such a crock. I never pay it. I once asked an associate editor at the Physical Review why I wold pay a voluntary fee and she told me "it should be part of your research philosophy". Fuck that: my research philosophy is not to waste precious funds, especially when it probably just goes to large "business" lunches for the editorial staff of a journal that is only the best because they say it is the best. I don't know if you use it but this is a very cool way to stick it to the journals.

[Victus]

Choobus wrote

I hate impact factors. IT's nothing more than a self fulfilling prophecy. It reminds me of bottled water. SOme journals (typically, the high imnpact factor ones) actually charge you for page submissions. Not only reprints, but you are encouraged to pay a "voluntary" fee (perpage) to "defray the costs of publication". It's such a crock. I never pay it. I once asked an associate editor at the Physical Review why I wold pay a voluntary fee and she told me "it should be part of your research philosophy". Fuck that: my research philosophy is not to waste precious funds, especially when it probably just goes to large "business" lunches for the editorial staff of a journal that is only the best because they say it is the best. I don't know if you use it but this is a very cool way to stick it to the journals.

:D

"Wait, wait. You want me to give you money so that you can attach your journal's name to my ground breaking experiment? Fuck you, and fuck the tree bark your journal is scribbled on", is our lab's response to such requests, I believe.