This is the last part of a four part posting, beginning with Everbody knows how reviews work
The question at the end is “Can you trust reviews.” I’d say yes you can – but of course you’ll be expecting me to say that.
I do hope, though, that I’ve given some fairly good reasons why the influence of advertisers is nowhere near as big as many people imagine it to be. The real issue is money – the simple need to actually make a living and pay for the roof over your head.
For a full time freelance, doing product reviews can be a pretty thankless task; as I mentioned, writing features can be a better way to make a living. I do a mix of the two – simple gadget reviews, plus features on specific areas about which I’ve decided to learn a lot of background (like digital TV), and related reviews.
Will an unpaid reviewer, or someone who’s not doing this to make a living do a better review? That really depends. If you’re doing it for love, and you have unlimited time to write 3000 words on a digital TV recorder, then good for you, and good for your readers. But I would dispute the assertion that an unpaid blogger must necessarily be more accurate and better informed than a professional writer.
A look around various blogs will find some that are, undoubtedly, excellent, written by people who are well informed, and spend time crafting detailed reviews. It will also find some that are shockingly partisan, and others that are barely literate.
On the whole, by virtue of employing editors and sub-editors, professionally written reviews will tend to read better; I don’t think that’s too contentious a point. Will they be more accurate? I’d say that they can be – where an editor commissions someone to write, because they have broad experience over many years, in a particular area, for example.
Are we all – both bloggers and professional writers – in thrall to PRs who let us keep shiny lovely gadgets? I don’t think so. Companies are far less generous than they used to be – and even in the past, most review kit that was particularly covetable was reclaimed, sooner or later.
Sure, I have cupboards with gadgets that time forgot – and that’s tended to be nothing to do with the review I wrote, but simply the march of progress. If a PR company wants back the broadband router than can’t even do ADSL2, they’re welcome – I need the space. It’s certainly not going to influence what I write about their next one.
If you think my living room is full of the latest AV gear, and fancy gadgets, it’s not. While many of us might take advantage of a press discount (though these days, you can often buy cheaper at places like Richer Sounds) if we’ve liked something, keeping a neat gadget just doesn’t tend to happen. Why would you write a good review of an indifferent product, just because you wanted to keep it?
Yes, we get to play with the latest fancy gadgets. No, on the whole, we don’t get to keep them. Sometimes, professional writers get to go on fancy trips – I went with Panasonic to their Convention in Munich earlier this year. I saw lots of new products, had a fun evening in a beer hall, and flirted with a cute PR guy.
Has it made me write better things about Panasonic products? Well, their TV came joint third in a roundup I wrote for Register Hardware, and I’ve been critical of other products they make, so if that was their plan, it doesn’t seem to have worked. Nor does that sort of thing, in my view, influence other writers I know.
I hope, at the end of this – admittedly rather long – piece, you’ll at least understand a bit more about how reviews work in the tech press, for the UK at any rate.
Of course advertisers have a role, but as I’ve explained, it’s a far, far smaller one than most people imagine.
Well, speaking as a freelance working for the same magazines you’re talking about since 1990…
1- I’m kind of astonished that you said you refused to use again a freelance who pitched a review for a new product from a company who let him keep the last product of that company’s that he reviewed for you. Unless it was a multi-thousand-pound piece of equipment, I’d say being allowed to keep stuff, at least for some months as a loan and often longer, was very common practice. At one time in the early 1990s, all the computer magazines I’m familiar with ran on “blagged” equipment and printed stuff out to send to production – the publishing companies did not install their own computer systems until much later. You gave stuff back if the companies asked for it, and you didn’t if they didn’t, and that was just hardware. No one ever returned software. There’s much less of this now, largely because computers have gotten so cheap.
2- You do not cover the practice of reviewing beta products, which I think can genuinely be deceptive. Software publishers want it, because they want the reviews out when the product is launched. But I remember vividly reviewed Packrat 5.0 in beta and writing a review for a now-defunct magazine saying that the product was so buggy and ill-conceived that it was not ready for release and would alienate the company’s long-standing users. The editor phoned me to say that I needed to tone the review down because you had to be kinder to beta products because their bugs would get fixed before release. It was my contention that the product was so buggy it *couldn’t possibly* be fixed before the release date. I was told to ask the company for a more recent release. They sent one. It was just as buggy. We argued…I can’t remember now how the review came out, but I can tell you this: when Packrat 5.0 was released it was just as buggy – and it killed the company.
3- You also do not mention the impact of press trips. Of course, there are many fewer of these these days (or I’m not invited on them) so it doesn’t really apply any more, but being taken to see stuff tended to mean that the companies and their products would get *covered*, (and reviewed) increasing their importance, even if the reviews and commentary were negative.
4- Also missing is the impact of NDAs. I think they’re pernicious, myself, because they enter the reporter into a cozy relationship with the company where the company gets to some extent to dictate what the journalist is allowed to report. So journalist feels warm and fuzzy because he’s been taken into the company’s confidence – but has surrendered control. I know people defend them on the ground that they need to know background in advance, and they need to be ready for new stuff when it comes out, but still, it bothers me, so I don’t sign them.
wg
Hi Wendy
With regard to your first point, there was a little more to it than that; essentially, I had good reason to believe from the way it was being pitched that things were not entirely on the level. But as you’ll appreciate, I don’t want to get into naming anyone; my main point is that, when there was a feeling that someone was trying to play us, we wouldn’t go along with it.
The other points are probably well worth addressing in another post – I didn’t intend this piece to go on for quite as long as it did originally; beta software and NDAs certainly aren’t helpful, and betas in particular are a real problem for reviewers.
NDAs, of course, aren’t the only way companies try to exercise control; after I first posted this piece, one person told me that they were invited recently to a press conference on the condition that there would be coverage, which I think is unacceptable.
Press trips certainly do feed into what I called ‘selection bias’ in the third part of the post; it’s a tricky one, because there is a clear impact from that point of view, but on the other hand, you can also get a lot of useful information, and background, from such trips, even if you don’t necessarily write anything about the trip or the products directly.
You make some good points, Nigel, and you make them well. But you neglect to mention the alarming rise in the quantity of advertorial reviews, which are totally different and generally untrustworthy. Not just because of what they contain, but what they don’t. I recently turned down good money to write a “sponsored feature” because the client specifically barred me from mentioning any competitors.
I often wonder just how many readers are aware that there’s a world of difference between a proper bit of editorial and an advertorial, sponsored feature, advertiser’s announcement, and all the various other deceptive names that these things tend to go under.
I’m comfortable writing under an NDA because it means I get to learn about things in time to think about them and – I hope – write a better piece. I don’t feel warm and cuddly about getting access – I just appreciate being able to write a piece without staying late in the office! With beta software we have to say it’s beta and judge it as beta; I became infamous with one product team for pursuing the question of a particularly problematic bug that was in an early beta. I think with these large-scale public betas that companies like Microsoft pursue readers actually have a better understanding because they’re often using the beta themselves.
I hate the word advertorial; as an editor, I label things ‘sponsored feature’ instead and make sure the design is different so it can’t be mistaken for editorial. How much difference that makes to the average reader, I’m never sure.
It’s almost universal for PRs to insist on guaranteed coverage before confirming invites to press trips these days (if we give you thins interview on this trip, who will you write it up for?). As a freelance writer that’s a bind; no editor is going to guarantee coverage of something that turns out not to be a story. But coverage doesn’t mean *good* coverage. I don’t know if the assumption that something is reviewed means it was worth considering is common. In my experience there aren’t many reviews that slam terrible products because there are too many products that have some value to cover to make room for the truly awful ones unless they’re really high profile, but that’s another factor in selection bias.
Sponsored features are awkward; as Mary said, the best you can do is make sure they’re clearly labelled, as ‘Sponsored Feature’ at the top of every page – I’ve had to do that as part of some of the ComputerActive Ultimate Guides that I edit. And, notably, the PCC has censured the Express for failing to do so, with some of the puff pieces they run.
I guess it depends on the context exactly what you feel comfortable mentioning or allowing not to be mentioned. For example, we had sponsorship from Lexmark for one of the ComputerActive Ultimate Guides, and that included a feature on wireless printing, clearly labelled. With the Lexmark logo all over it, it would have looked a bit peculiar if it were to talk about other brands of printer, rather than extoll the virtues of those from Lexmark, so I’m happy to allow that, and I don’t think readers would have thought it was anything other than a Lexmark piece.
Similarly, we ran one for Devolo home plugs; it’s a case of striking a balance, really – the copy explained there are other types of HomePlug, mentioning specific types that aren’t made by them, and covering the technology in general. But where products were mentioned, they were the sponsors.
Again, I don’t have much of a problem with that – but I would have if they said “don’t mention any types of HomePlug device that we don’t make” or “always refer to it as devolo HomePlug.” The former is misleading by omission, and the latter is misleading to a degree, but also just makes something read really badly.
I think there’s also possibly a difference depending on how the advertorial is commissioned; if a company asks you to write it, with a view to placing it somewhere, you don’t necessarily know how it’s going to appear, and if it’s going to be clearly marked as advertorial/sponsored, or whatever. And as a freelance, that’s also a bit of a risk (and why some don’t put their name to such thing).
With the Ultimate Guides, we’ve tended to commission the copy ourselves for this sort of thing, which means that we can make sure it doesn’t just read like something spewed out by their marketing department – though I suppose some would argue that at least in the latter case, surely even the most careless readers will spot that they’re being spun a line.
Where it clearly crosses the line that Robert talks about is where sponsored/advertorial copy appears, but with nothing to distinguish it from the other content; that is deliberately misleading readers, but I don’t think – at least in the UK tech press – it happens. Despite the efforts of PRs who helpfully offer features by email from time to time.
Been on the press trips, been fed, been watered for the last 20 years and I still happily slag off the products. If it’s a crap product it’s a crap product, and no amount of free booze will change my mind.
Part of the problem in the past was the space available a 1/2 page review of old allowed you to just about describe the product, I once had to do a 1/2 page review of an Ofiice-like product in 250 words, which was madness. It’s no longer a problem with the Internet, indeed the longer the review the more page views you get and the more opportunities there are to put ads next to the copy.
To be honest time frames don’t really matter either. If you’ve been reviewing products for five years you just know if a product is good or bad. Chances are you have looked at all of the previous productsfrom that company and looked at all of the competitors products, so you know a good product when you see one. People like Guy Kewney could review a product in 30 minutes and he’d find more errors in those 30 minutes than most reviewers would find in weeks of looking.
There’s also a lot of expertise in this field, we’re not all english graduates. I’ve got a degree in computing, and a masters in Microprocessor engineering and I was a programmer for 8 years before I became a journo. I know this stuff in and out.
I’ve also have never been offered cash for a good review. I have however been physically threatened by a business owner because they thought a review I did was unfair. They also threatened to remove their advertising unless I explained in detail why I thought their product was crap, which I did. They ended up apoligising, kept advertising in the mag, and I lived to review their next product.
If anything is causing problems it’s the rates on offer The rates are unrealistic. I remember commisioning people like Wendy back in the early ’90s and the rates just haven’t changed much since then. You can’t live on the rates offered thses days. I was offered £25 to do a full page review of a product by a website a few years ago, that’s a tenth of the going rate of 20 years ago.
Aren’t the time frames pretty closely linked to the question of the rates? The reason it’s hard to make a living on the rates offered is because of the time it takes. If you allow the time to do a really thorough review, you’re only making minimum wage; you might do a bit better on those few products where you can bash out several hundred words, and be sure you’ve explored it thoroughly in a morning, but I contend that those products – even for the best of us – are few and far between.
That’s especially true now, where there can be so many options, or possible interactions with other software, drivers, and so on. You can seldom say, with hand on heart, “you won’t have any problems with this” because chances are, it simply isn’t economical for you, as a reviewer, to try everything out on Windows XP, and Vista, and 7, or to see if the web interface works reliably in IE, Firefox and Opera, or any one of those other little gotchas that can crop up.
You can do some of them, and if you’re an experienced reviewer, you’ll look for the things where you know similar products fall down. But the complexity of much modern kit means that, even then, you’ll have a pretty long list to work through