Part 2 – The problems with reviews
You might – and I hope you will – accept by now that the finances are one of the main reasons for some reviews not being as good as you might like them to be. But still, there are questions about those pesky advertisers. Do they get more reviews than people who don’t, or better ones?
In the case of freelance reviews, the freelance simply won’t have a clue whether or not a company is advertising in a magazine, so it’s hard to see how that can affect their conclusions – and I don’t recall meeting enough freelances claiming “X changed my review and gave the product a higher rating” to make me think that happens systematically. Yes, a rating might occasionally be tweaked by an editor, but that’s often because the editor’s got knowledge of other comparable products, or feels the text doesn’t match up with the rating given.
No one has ever, in all my years, called up and said “Can you review this, and go easy on them,” let alone suggesting it’s because they’re an advertiser.
Even with in-house reviews, it doesn’t really happen. There’s not as much contact between advertising and editorial staff as some people imagine there to be, and a staff writer who’s told by the reviews editor “350 words on this, by Monday please” really isn’t likely to have a clue whether or not the product is from a potential advertiser or not. And again, in my experience, the commissioning editor doesn’t give the writer a line on what they want the review to come out like. I have been asked precisely once in almost 20 years to alter the tone of a piece – and that was to make a vox-pop piece more critical of a company, not less, by selecting different quotes.
Hang on, someone at the back is doubtless shouting, but don’t people who advertise win more awards? Impossible to say, with any certainty, and you run into a case of “post hoc ergo propter hoc,” which is to say that you can’t necessarily prove a causal link.
If a company advertises, and wins awards from a magazine, which came first? Do they advertise more in a magazine because they have an “Editor’s choice” logo that looks like an endorsement? Perhaps. Did they get that because they advertise? Most unlikely, in my experience, that there was a causal link.
However, I do accept that there is what could be called a ‘selection bias’ at times. That is that when someone on the ad sales team is speaking with a client, who mentions they have a new product, they’ll pass on the name of whoever commissions reviews. And, sometimes, when the reviews editor is hoping to get a product in from a company that they’ve not written about before, they might ask the advertising team if they have a contact name.
But does that ever extend to the ads team saying “can you do a good review of this? It’ll help us sell some pages” Again, in my experience the answer is no. There is a ‘selection bias’ in that contact with the ads department might make it more likely that the reviews editor knows of a product or company, and makes it stand out a little more from the mass of press releases vying for their attention.
Even then, that’s a long way from garnering a good review, just because someone advertises. As I mentioned before, in my experience, publishers will back an editor who upsets an advertiser.
Selection bias isn’t ideal – but it’s probably unavoidable, and of course it takes other forms too. As a reviews editor, you’ll get to know the various PR people, and you’ll get on better with some than with others. You’ll learn which ones can reliably get you a product first, or at short notice. And when someone else lets you down, you know that if you call up a particular company, they’ll be able to send you their latest PC or gadget, so that you can fill the half page gap that’s been left by someone else’s product being stranded in a volcanic ash cloud.
