What Makes a Good Study?

What Makes a Good Study?

We’ve all seen news segments and click-bait headlines touting a “new scientific study” containing alarming health claims that make us question if we should avoid certain products or change our daily behaviors. (We’ve touched on this topic before). Some of these reports are grounded in sound science and are worthy of our time – the American Cancer Society’s recent recommendation lowering the suggested age for colonoscopy screenings from 50 to 45 is one example. But in today’s report-first / clarify-later (or never) news environment, many studies that are utterly worthless gain national attention – because the views they generate are simply too tempting for news organizations to fact check. And when news outlets blindly cover these “junk” studies without determining if they have any redeeming scientific qualities, they do a great disservice to the American public.

Nearly 50% of all studies the news media report turn out to be wrong. And when a more robust study on the same topic reaches a different conclusion, the facts rarely receive the same level of exposure as the initial faulty coverage. That gives readers and viewers little reason to question the legitimacy of bad studies, which in turn incites baseless public fear and leads many to make unnecessary changes to the way they lead their lives.

Elements of Good Research

The characteristics of a credible study aren’t complicated: They must be rigorous, thorough, and replicable. But many studies today fall well short of those standards and make observational claims with small sample sizes where no reliable scientific conclusions can possibly be drawn.

Statistical Significance

Statistically significant and clinically important results exhibit important, meaningful differences between study subject groups. If a study does not have a large enough sample size, it could fail to detect an important difference between subject groups. If a study has enough participants, even the most trivial differences between groups might indicate a statistical trend, but the data doesn’t necessarily mean these differences are clinically important or meaningful to normal people. The best practice is not solely about selecting a certain sample size – it’s the extent that the researcher goes to eliminate doubt and to build trust in the results by employing the best methods of study design.

Methodology

Trustworthy studies (1) rely on epidemiological research, assuming the sample sizes are sufficient and follow the study subject group practices noted above (2) use meta-analysis (3) follow a peer-review process or (4) apply a randomized controlled trial, the gold standard. In public health, epidemiologists are taught to look at the probability value (p-value) to determine if the observed differences between subject group outcomes (i.e. ill and not ill) are true differences, or if the observed outcomes are simply a result of chance. But many academic researchers don’t take the time to use rigid epidemiological methods, instead making far-reaching conclusions based on limited evidence or small numbers of people. Some even engage in data manipulation to achieve dubious statistical outcomes (this practice is called p-hacking, you can read more about it here). According to the NIH, a best practice for all researchers would be to explain why the claimed relationship may indeed be real, and not a rare fluke or misuse of data analysis methods. For example, demonstrating why results agree with other studies in different human populations would further clarify statistical value. Clinical researchers are also encouraged to consider other relevant measures because of the severe limitations on patient sampling and hypothesis testing.

Peer-Review

Peer-review is one of the most important processes in scientific research. In the best case, peer-review strengthens quality control on studies before they are released to the public. For that reason, trustworthy scientific journals require all submissions to be peer-reviewed. But not all peer-reviews are equally rigorous. A good peer-review should be critical and performed by qualified researchers who do not share affiliations or ideological biases with the author of the study being reviewed (you can read more about what makes a good peer-review from the American Chemical Society here). Unfortunately, researchers often fail to use objective peer-reviewers and opt for those who are already predisposed to support their conclusions.

Junk Science in Action

Vinyl material is often the target of this kind of shoddy science and reporting. Activists, agenda-driven researchers, and some journalists often promote poorly designed and executed studies that malign vinyl. Because vinyl products and manufacturing processes are complex and highly technical, most readers aren’t able to sort fact from fiction in these studies. Here are a few recent examples:

New York Times on ‘chemicals in your mac and cheese.’ In 2017, the New York Times ran a story based on an observational study that purported to identify toxic chemicals in mac and cheese from vinyl containers. Just a few problems: the study doesn’t actually say anything about the effects of those chemicals, it just assumes that they’re inherently toxic, which isn’t true. Second, the study invents a new standard for exposure which contradicts the scientifically accepted standard. The reporter didn’t note either of these facts even though they were provided to her before publication. You can read more here: NYT’s Rabin Compromises Journalistic Integrity In “Click-bait” Clown Story On Phthalates

ABC News promotes a flawed study on phthalates and language delays. In 2018, an ABC affiliate in Duluth, MN ran a segment about a study that claimed exposure to vinyl products could cause language delays in children. The study in question was an observational study with a minuscule sample size of only about 1,200 people (a small sample for this type of study), upon which it based a far-fetched cause-and-effect relationship. Additionally, the study relied on self-reported questionnaires which are highly unscientific. The reporter failed to provide any of this important context for viewers; in fact, she didn’t even identify the study so that viewers could verify her report. More about this report here: Correcting The Record On Vinyl And Phthalates

Environmental Health News article on phthalates and motor skills. Most recently, Environmental Health News published an article about a study on phthalate exposure and vinyl material. EHN starts off with the assertion that “[k]ids exposed to phthalates prenatally and as 3-year-olds have decreased motor skills later in their childhood.” They don’t acknowledge until the second-to-last paragraph that the study doesn’t actually prove that. And once again, the study observed only 209 kids; a sample size far too low to draw such overbroad conclusions.

Why Good Science Matters

When reporters allow sensational findings of flawed studies to drive their decision-making in what they relay to the public, they spread confusion and promote inaccurate information that can have real-world consequences. One only has to look as far as the child vaccination issue to understand how widely-reported junk science can lead people to make poor decisions that can impact public health and safety. For years, observational studies claiming a link between the MMR (measles) vaccine and autism dominated the headlines, leading many parents to avoid immunizing their children. As a result, measles outbreaks are now on the rise. But, landmark research on the MMR vaccine which was recently completed that included 657,461 children studied over 20 years in a randomized control trial (the most rigorous standard in study design) found no link between the vaccine and autism. And as expected, the attention this study received paled in comparison to the misleading reports from the past decade.

Media responsibility matters. And when journalists reflexively publish click-bait stories based on studies they know have no scientific credibility, they willfully deceive readers and viewers. It doesn’t take much to ascertain whether a particular research report passes the red face test.

It just requires the reporters who cover them to care.

MORE: Scientific Studies, Last Week Tonight

 Dr. Oz (Once Again) Misleads Public: Pushes Irresponsible Claims On Vinyl Pool Toys

Dr. Oz (Once Again) Misleads Public: Pushes Irresponsible Claims On Vinyl Pool Toys

Dr. Oz – like all journalist physicians – has a duty to act as a filter and evaluate the credibility of the studies he covers before promoting them to a national audience. And when certain studies fall well short of even the most rudimentary scientific standards, he has a responsibility to his viewers to skip covering them to avoid inciting unnecessary public hysteria. 

But he failed that test this week when he devoted more than seven minutes on his national show to an obscure German study on vinyl pool toys, the findings of which even he and the guests on his show agreed were inconclusive.

Dr. Oz clarified in the segment that parents shouldn’t be alarmed by the study’s uncorroborated assertions. But the time and attention he dedicated to this relatively unknown German study did exactly that in the minds of many of his viewers.  His show even featured alternatives to vinyl pool toys, giving viewers a false reason to believe some of the study’s findings may indeed be worthy of concern.   

Are we surprised? The British Medical Journal determined that over 50% of the recommendations on Dr. Oz show were either flat wrong or based on no scientific evidence whatsoever. And he maintained that dubious batting average by irresponsibly airing this segment featuring a slew of baseless claims about the safety of vinyl pool toys and PVC products, despite our best efforts to enlighten him of the facts in advance.

PVC’s safety has been proven time and time again by countless studies and decades of real-world use. It’s one of the most widely researched materials in the world. It’s manufacturing complies with some of the strictest regulations of any material produced in the United States. And its safe use has been verified over and over by U.S. government agencies tasked with protecting the health and safety of the American public.

But Dr. Oz opted to incite unnecessary fear about vinyl.  He promoted an obscure German study that made a number of unsubstantiated and irresponsible claims about vinyl. This isolated, unreplicated study uses flawed methodology to make sweeping claims about an entire category of vinyl pool toys based on a review of only four product samples. And it asserts that odor emissions detected from those products – at any level – can indicate toxicity to humans, which just isn’t true. Any reputable scientist wouldn’t give these inconclusive results the time of day, let alone promote it to millions of people -- or encourage people to change their daily lives -- because of it. Yet that’s exactly what Dr. Oz did.

We told Dr. Oz prior to his segment that no credible scientific findings could be drawn from the German study -- but he went ahead with his segment anyway. We sent him a lengthy response highlighting the many reasons why the German study falls short of any definitive scientific conclusions. We pointed out that the study relies on data gathered using only four products—an insufficient sample size to justify the claims it makes. We also noted that two of the samples were from unidentified origins, and one was sourced from China. This is important since PVC pool toys sold in the United States are required to follow strict production regulations that other countries may not observe. And we clarified that the authors of the study themselves noted that their findings were inconclusive and would need additional research to prove. In the two years since the study was published, that research has never been done.

Dr. Oz and his guests even noted that the study’s findings were inconclusive - but the show went ahead with the segment anyway. 

  • Mara Schiavocampo, a correspondent for Dr. Oz featured in the pool toy segment, stated:“... we have to keep in mind that this is a very, very small study. They only looked at four toys. Even the researchers of this study said, ‘we need to study it more.’ So the takeaway here is not that these pool toys will make you sick, it’s just that we need to look into this more.”

  • Toxicologist Tamykah Anthony stated, “I don’t want to alarm a lot of people because the study was kind of small.”

  • And even Dr. Oz himself said, “Good advice for everybody is don’t be scared of the pool floats...” 

Yet Dr. Oz ran this baseless fear-mongering segment anyway, because the opportunity to scare viewers and generate clicks to his website was simply too tempting to pass up.

The chemicals that the study claims are emitted by PVC pool floats aren’t even used in the production of PVC. They are often used in the solvents that adhere ink to the floats, but they are not used in PVC or vinyl material. We pointed this out to him as well before his story went to air, but he ignored this, too.  What’s more, Toxicologist Tamykah Anthony claimed that “PVC plastics” are “endocrine disruptors” when there is no credible evidence to support her statement.

Dr. Oz’s decision to push this uncorroborated German study fits with his track record of peddling bad science. As we noted at the outset, a review by the British Medical Journal found that more than half of Oz’s advice wasn’t supported by scientific evidence. His claims have been ridiculed in the Washington Post, New Yorker, and Popular Science, among other outlets. The American Medical Association Journal of Ethics expressed serious concern about Dr. Oz, and physicians at Columbia University told CNN they were “dismayed” when he joined their faculty. He’s even been called before Congress to answer for his promotion of dubious diet supplements.

Some of the segments presented by Dr. Oz are downright laughable. In the past, he’s claimed that eggplants can cure cancer, that numerology holds secrets to health, and that people should eat pasta and broccoli for breakfast. Here’s a supercut of some of his “miraculous” claims from the Washington Post:

So it’s no surprise that he’s promoting a questionable study on vinyl pool toys. Dr. Oz succumbed to click-bait temptation, once again, and put his own self-interest ahead of his viewers. He pushed a flawed scare narrative that will no doubt generate views across his platforms and earn him money from his corporate sponsors, but won’t help the public live healthier lives.

Facts help people.  Regardless of how boring they can sometimes be, they matter.

And when personalities like Dr. Oz push fear-mongering over facts, in pursuit of fame and financial success, the public loses.

Vinyl Verified Responds to “Flawed,” “Irresponsible” Australian Ocean Water Study

Vinyl Verified Responds to “Flawed,” “Irresponsible” Australian Ocean Water Study

A recent study from Australia, the findings of which are based on a flawed premise that PVC is a common ocean plastic, has lured certain national media outlets to promote its reckless and inaccurate claim that PVC dangerously affects phytoplankton productivity in our oceans. These allegations are wildly misleading – and the media outlets that cover them do a great disservice by perpetuating unreliable information to unwitting readers and viewers.

Specifically, the study’s authors erroneously make a key assumption on a material that is not commonly found in our oceans. They draw irresponsible conclusions that are impossible to replicate in a real-world ocean environment. And they fail to use a level of scientific rigor that is anything close to the gold standard.

First, the study suggests PVC is a common material in our oceans – and that simply isn’t true. Of all of the plastic found in the world’s waste stream, just 2.8% is PVC according to the United Nations. And, there is no reason to believe that PVC would end up in the ocean at all, since it would be less likely to float on the rivers that carry plastic waste to the ocean in the first place. The study bases its conclusions on the false premise that PVC makes up a large portion of the plastic in our oceans – any such claim is categorically false.

Not only were typical ocean plastics not tested, but only two plastic items were also selected and tested, by the investigators. The authors went to a local shop, bought a flexible PVC mat, carried it in a plastic bag, cut them both into pieces, immersed them into artificially created ocean water, and conducted testing to draw their results. They made a claim about an entire category of materials based on this hyper-selective and narrowly-focused approach, without any consideration for why the sample was selected, and how it would apply to the projected conclusion that an average reader would take away. For these reasons, the study cannot be trusted.

The study falsely suggests that PVC is a major contributor to the destruction of our marine ecology. The results of the study determined that zinc is the primary harmful contaminant identified. While it is used as an additive in some vinyl products, zinc is a far more common element in countless materials and products specifically used in or around our shorelines, including fencing, metal piping, steel ships, and notably, sunscreen. Furthermore, over 35,000 tonnes per year of zinc come from volcanic releases into the environment. Based on the study’s findings, to suggest that PVC could have a major influence on our oceans is wildly irresponsible.

The outlet that published the flawed research did not appear to follow established NIH’s guidelines in assessing the study’s credibility. The publication, Communications Biology, even acknowledges on its website that its mission is to deliver the “rapid dissemination of results” – which stands in direct contradiction to its aspirational claim to be a source of “high quality” research. The National Institutes of Health (NIH) specifies the comprehensive review process studies must undergo in order to be deemed credible. Readers are right to question whether Communications Biology followed these guidelines, as any outlet that prioritizes speed-of-delivery of the research they publish is likely to fall well short of the NIH’s high standards.

Readers deserve to know when scientifically sound studies raise legitimate public health and environmental concerns. Society also deserves to have information from reliable sources that collaborate and engage with stakeholders to ensure the highest quality outcomes, and to anticipate future research needs. They don’t deserve to be exposed to unreliable, uncorroborated research that only sparks baseless public fear.


The Winchester Star Misleads Readers on Vinyl Windows

The Winchester Star Misleads Readers on Vinyl Windows

Media outlets have an obligation to the readers they serve to publish balanced stories on the issues they cover. When they don’t live up to these standards, and only present a one-sided view, they – at a minimum – have a responsibility to publish an opposing perspective, typically in the form of a letter-to-the-editor, so that readers have all of the facts, and can decide for themselves who to believe.  

The Winchester Star failed on both counts.

Last month, the Star ran an article about building code changes for Old Town Winchester that included several false and misleading quotes about vinyl windows from Kevin Walker, the Chairman of Winchester’s Board of Architectural Review (BAR).

Susan Wade from the VInyl Institute (VI) submitted a letter-to-the-editor to the Star correcting Mr. Walker’s claims to Adrian O’Connor, the Star’s editorial page editor.

Adrian O’Connor never responded.

We followed up.

Still, no answer.

We made every effort to allow the Star to correct the record. They ignored us.

There was a time years ago when media outlets held all of the cards. Reporters were able to say anything, or quote anyone, about PVC material, where little regard was given to providing readers accurate or balanced information on vinyl issues. And when media outlets ignored attempts to correct published misstatements via letters to the editor, like the case here with the Winchester Star, industry had little recourse to make sure the facts prevailed.

Not anymore.  With the Vinyl Verified platform, we can do it ourselves.  It’s why the recent interaction with the Winchester Star (or lack thereof) further justifies the work we do to expose media bias, when it occurs, about vinyl material.  

We believe Star readers have a right to know the truth about vinyl windows. Since the Star declined to present a balanced view of the facts, in the spirit of transparency and accuracy, we are posting the letter the Star didn’t want you to see that corrects Mr. Walker’s erroneous claims.


What’s Old Sometimes Isn’t New Again … And That’s AOK

By Ned Monroe, President and CEO of the Vinyl Institute

It’s perfectly reasonable for historic city districts, like Old Town Winchester, to require the use of period materials in new construction and existing building repairs. But it’s irresponsible when City officials who oversee those projects publicly distort the facts and mislead citizens about modern materials and products, such as vinyl windows, in the process.

The Chairman of Winchester’s Board of Architectural Review (BAR), Kevin Walker, did just that in a recent article. He claimed vinyl window manufacturing is “highly toxic” – yet if he ever visited a production facility, he’d know that the industry operates under some of the strictest regulations in the world to ensure workplace safety.  

Advancements in building materials, including vinyl, have brought enormous benefits to homeowners and builders alike. Vinyl windows are lightweight, easy to install, provide unsurpassed durability, are cost-effective, and can be customized in their design to accommodate anyone’s taste.

Historic projects should use materials of their time. And if Mr. Walker would prefer to use windows with lead counterweights in his home, that’s entirely his prerogative.

But it doesn’t give him the right to publicly disparage other attractive, modern-day materials, like vinyl windows, which provide practical, reliable, and affordable options for homeowners.

Firefighters Deserve Facts on Environmental Health

Firefighters Deserve Facts on Environmental Health

 

Our nation’s firefighters have earned the right, without exception, to know what the potential environmental health risks are that impact their community. As with any important public health issue, it’s critical the facts prevail, and that credible science guides the debate in determining potential actions necessary to ensure their safety.  

Firefighter leaders, rightfully, are taking an active role on these important issues. As a representative voice of thousands of courageous men and women, they play an important part in raising awareness for firefighter health and advocating to fix the root cause of the health problems confronting this community.

Recently, however, some firefighter leaders have associated firefighter health issues with factors that have not been credibly linked. They have singled out the use of plastic piping in buildings, including PVC, claiming that emissions from these materials create a greater risk to firefighters when they’re burned, despite scientific evidence showing other causes may be to blame.  

Facts show that all combustible materials, including wood, will yield toxic and carcinogenic byproducts when involved in an accidental fire. And the evidence points to other causes – including diesel engine exhaust – as contributing to cancers in firefighters.

What’s more, certain metallic alternatives to vinyl piping actually increase the risk of flame spread.  Ductile iron pipe, for example, has an asphaltic coating that’s highly flammable.  By contrast, PVC pipe doesn’t spread flames, not to mention it is lighter weight, far easier to use, doesn’t require welding – and outlasts competing materials.

That’s why PVC pipe has been the material of choice for builders in residential and commercial construction for decades, because it’s known for delivering safe, reliable and affordable service.

Firefighter leaders have a unique opportunity to speak out, call for action -- and work passionately, and collaboratively, to help fix firefighter health problems. And they have every right in the world to demand answers for their colleagues who face dangerous conditions each and every day on the job. 

Likewise, those on the front lines deserve to have the facts regarding the environmental risks they face in order to best protect their health and safety.

 

Topeka, Kansas: A Case Study in Pipe Durability

Topeka, Kansas: A Case Study in Pipe Durability

The Topeka Capital-Journal recently published a story on the large number of water main breaks that the city has endured in recent years. Topeka’s water system relies on 870 miles of pipe. Since the 1940s, Topeka has diversified its system by adding ductile iron pipe and polyvinyl chloride (PVC) pipe. Today, the system is 39% cast iron pipe, 27% ductile iron pipe, and 25% PVC pipe (figure 1).

 

Percentage of Pipe Materials Used

Figure 1: Topeka's system of pipes is about 39% cast iron, 27% ductile iron, and 25% PVC, with the remaining made of various other materials.
 

Of the city’s water main breaks, approximately 66.8% were from cast iron pipes, 18.4% were from ductile iron pipes, and a mere 4.5% were from PVC pipes (figure 2).

Percentage of Pipe Breaks by Material

Figure 2: In Topeka, 66.8% of breaks occur from cast iron pipes, 18.4% from ductile iron pipes, and 4.5% from PVC pipes.

In Topeka, all iron pipes have a failure rate that is more than 18x the failure rate of PVC (figure 3). Ductile iron pipe devotees (and their lobbyists) will tell you that ductile iron is a great improvement over cast iron. And yet, ductile iron pipe still fails at more than 4x the rate of PVC pipe (figure 4).

Pipe Breaks: Iron V. PVC (%)

Figure 3: All iron pipe accounts for more than 18x as many breaks as PVC pipe.

Pipe Breaks: Ductile Iron v. PVC (%)

Figure 4: Ductile iron pipe accounts for more than 4x as many breaks as PVC pipe despite the fact that ductile iron pipe makes up almost the same share of the system (27%) as PVC pipe (25%).

In fact, Topeka has seen a significant number of ductile iron pipes grossly underperforming their expected life span:

Ductile iron pipe should have a lifespan of 100 years, but [Deputy Director of Utilities Braxton Copley] said the city has seen a large number of breaks in pipes that are only 40 to 50 years old.
— Topeka Capital-Journal

The ductile iron pipe industry wants the public to believe that its product is more reliable than PVC but the facts show otherwise. As demonstrated by this case study, PVC is clearly the more durable material.

Living Building Chronicle Perpetuates Activist Rhetoric on PVC

Living Building Chronicle Perpetuates Activist Rhetoric on PVC

A recent piece by Living Building Chronicle, published by the Kendeda Fund, contains a series of broad mischaracterizations about PVC that require correction.  What’s interesting is that the author, Ken Edelstein, goes to great lengths to promote a PVC material competitor, raising reasonable questions regarding a possible hidden motivation driving this story. Regardless, we’ll take the opportunity to ensure the facts are reflected for the record.