Problems with Research
In pursuing my master’s degree, I have become more perceptive to numerous forms and sources for research, both academic and less scholarly. While consuming these materials, I have become significantly more critical of the sources of my information for my academic purposes, as well as my non-academic learning. While I am by no means an expert on scientific research, there are a few examples I have recently discovered that stand out as surprisingly misleading or downright wrong.
Lack of a Sufficient Control Group
Having recently cracked down on my diet and exercise habits, I started incorporating chia seeds into what I eat. They mix well into a number of different foods, and pack an objectively impressive dietary and nutritional punch. Lots of fiber, lots of protein, and relatively few calories. Investigating further, I learned that some (https://runnersconnect.net/chia-seeds-running/) runners use chia seeds as a means of carb loading for long distance events. The Journal of Strength and Conditioning Research even (https://www.ncbi.nlm.nih.gov/pubmed/21183832) published a study on the effects of chia seeds for long distance runners.
The odd thing about these studies, however, was that they seemed to compare 100% gatorade to 50/50 gatorade and chia seeds for the purpose of carb loading. While that’s all well and good, it seemed to lack a true control group. In my mind, a control group for this study should observe runners without any sort of carb loading techniques, and then compare the gatorade and chia seed techniques to that. Sure, there was no statistical difference between the performance of runners using just gatorade versus the gatorade and chia seed mixture. But what about runners who didn’t use anything at all. Did either form of carb loading have any effect on the runners?
Maybe the control of 100% gatorade is sufficient for the study in question. Admittedly, adding in a group without any sort of carb loading technique broadens the scope of the study considerably. No longer is the study simply comparing two different techniques of carb loading. It begins to question the validity of carb loading at all by adding a group without that method.
At the same time, the study feels biased in favor of traditional runner techniques. It doesn’t even seem to acknowledge the possibility that removing the gatorade altogether would have no significant effect on the performance of the test subjects. Maybe this was considered in the full text of the study, and there could be a sufficient reason for the exclusion. With access to just the abstract, I can’t fully analyze how thoroughly they considered the possibility of removing carb loading practices entirely. With the information that I have, however, I genuinely feel as though the researchers left out a genuine control group, and thus reduce the validity of their findings.
RapidBI. (2016, April 30). How to write an internal communications plan and strategy. Retrieved December 16, 2017, from https://rapidbi.com/write-internal-communications-plan-strategy/
Lack of a Sufficient Control Group
Having recently cracked down on my diet and exercise habits, I started incorporating chia seeds into what I eat. They mix well into a number of different foods, and pack an objectively impressive dietary and nutritional punch. Lots of fiber, lots of protein, and relatively few calories. Investigating further, I learned that some (https://runnersconnect.net/chia-seeds-running/) runners use chia seeds as a means of carb loading for long distance events. The Journal of Strength and Conditioning Research even (https://www.ncbi.nlm.nih.gov/pubmed/21183832) published a study on the effects of chia seeds for long distance runners.
The odd thing about these studies, however, was that they seemed to compare 100% gatorade to 50/50 gatorade and chia seeds for the purpose of carb loading. While that’s all well and good, it seemed to lack a true control group. In my mind, a control group for this study should observe runners without any sort of carb loading techniques, and then compare the gatorade and chia seed techniques to that. Sure, there was no statistical difference between the performance of runners using just gatorade versus the gatorade and chia seed mixture. But what about runners who didn’t use anything at all. Did either form of carb loading have any effect on the runners?
Maybe the control of 100% gatorade is sufficient for the study in question. Admittedly, adding in a group without any sort of carb loading technique broadens the scope of the study considerably. No longer is the study simply comparing two different techniques of carb loading. It begins to question the validity of carb loading at all by adding a group without that method.
At the same time, the study feels biased in favor of traditional runner techniques. It doesn’t even seem to acknowledge the possibility that removing the gatorade altogether would have no significant effect on the performance of the test subjects. Maybe this was considered in the full text of the study, and there could be a sufficient reason for the exclusion. With access to just the abstract, I can’t fully analyze how thoroughly they considered the possibility of removing carb loading practices entirely. With the information that I have, however, I genuinely feel as though the researchers left out a genuine control group, and thus reduce the validity of their findings.
Number Conversions
I frequently see number conversions in sources. Usually, these conversions are between different units, frequently imperial to metric or vice versa. Occasionally, they may convert a figure to a more colloquially anticipated phrasing. In either case, I expect these conversions to be equivalent. To err in conversion gives me the impression that the author didn’t proof their work, as well as introducing doubt in which figure is the correct one.
“According to research from Gallup, 69% of employees are either not-engaged or actively disengaged on the job. Further research from the organization estimates that over £185,000 million (£0.19 Billion) is lost annually due to lower productivity from actively disengaged workers alone.”
I encountered the above conversion while doing research for a paper. The study indicates financial loss from decreased productivity. However, the two figures aren’t equivalent. To make the contrast more obvious, I’ll remove the verbal qualifiers from both. “£185,000 million” would be £185,000,000,000. The other quoted figure of “£0.19 Billion” converts to £190,000,000.
Even besides the strange change in significant figures, these values are 3 orders of magnitude different. Which is it? As much as this faux pas is unacceptable enough, the reader might be able to discern the intended value from the linked research. Nope, the link goes to a completely unrelated page that makes no mention of the financial loss of decreased productivity.
So this article managed to get a number wrong by 3 orders of magnitude, AND the link they offered gave no more clarity to the issue. How many other links on the page fail to address their context? Is anything on the site accurate or backed up with science. It is at this point that I write off a source as unreliable and move on to the next item in my search results. I’d advise you to do the same.
If you want your content to appear accurate and reliable to your audience, maybe try to include a valid control group and get your number conversions right. Even if there are valid reasons behind your decisions, explain them in a way that someone outside your field would understand. Maybe maintaining a carb loading method of gatorade would make sense to those inside the endurance exercise field would understand, but it doesn’t make sense to me. Maybe “£185,000 million” and “£0.19 Billion” are equivalent in some strange world (they aren’t), but if you don’t explain how, it makes me severely doubt your credibility.
Resources:
Illian, T. G., Casey, J. C., & Bishop, P. A. (2011). Omega 3 Chia Seed Loading as a Means of Carbohydrate Loading. Journal of Strength and Conditioning Research, 25(1), 61-65. doi:10.1519/jsc.0b013e3181fef85c
I frequently see number conversions in sources. Usually, these conversions are between different units, frequently imperial to metric or vice versa. Occasionally, they may convert a figure to a more colloquially anticipated phrasing. In either case, I expect these conversions to be equivalent. To err in conversion gives me the impression that the author didn’t proof their work, as well as introducing doubt in which figure is the correct one.
“According to research from Gallup, 69% of employees are either not-engaged or actively disengaged on the job. Further research from the organization estimates that over £185,000 million (£0.19 Billion) is lost annually due to lower productivity from actively disengaged workers alone.”
I encountered the above conversion while doing research for a paper. The study indicates financial loss from decreased productivity. However, the two figures aren’t equivalent. To make the contrast more obvious, I’ll remove the verbal qualifiers from both. “£185,000 million” would be £185,000,000,000. The other quoted figure of “£0.19 Billion” converts to £190,000,000.
Even besides the strange change in significant figures, these values are 3 orders of magnitude different. Which is it? As much as this faux pas is unacceptable enough, the reader might be able to discern the intended value from the linked research. Nope, the link goes to a completely unrelated page that makes no mention of the financial loss of decreased productivity.
So this article managed to get a number wrong by 3 orders of magnitude, AND the link they offered gave no more clarity to the issue. How many other links on the page fail to address their context? Is anything on the site accurate or backed up with science. It is at this point that I write off a source as unreliable and move on to the next item in my search results. I’d advise you to do the same.
If you want your content to appear accurate and reliable to your audience, maybe try to include a valid control group and get your number conversions right. Even if there are valid reasons behind your decisions, explain them in a way that someone outside your field would understand. Maybe maintaining a carb loading method of gatorade would make sense to those inside the endurance exercise field would understand, but it doesn’t make sense to me. Maybe “£185,000 million” and “£0.19 Billion” are equivalent in some strange world (they aren’t), but if you don’t explain how, it makes me severely doubt your credibility.
Resources:
Illian, T. G., Casey, J. C., & Bishop, P. A. (2011). Omega 3 Chia Seed Loading as a Means of Carbohydrate Loading. Journal of Strength and Conditioning Research, 25(1), 61-65. doi:10.1519/jsc.0b013e3181fef85c
RapidBI. (2016, April 30). How to write an internal communications plan and strategy. Retrieved December 16, 2017, from https://rapidbi.com/write-internal-communications-plan-strategy/
Comments
Post a Comment