Under The Bus

3rd February 2015

I hadn’t expected to be returning so soon to the subject of the Saatchi Bill consultation. But after I wrote my post criticising how their actions had undermined that consultation, the Saatchi bill team have responded. Yes indeed, and what a response it is.

I’m going to assume you’re familiar with the bill, and with the arguments in my previous post on the consultation. However, as an aid to recall I’ll do a quick recap of the consultation timeline and provide links as I go along to where more detail can be found.

Their, er, “rebuttal” is here. They haven’t troubled themselves to link to my post as they attempt to rebut it, but I do the courtesy, as I always do, of linking to theirs. I want you to be able to check for yourself whether I am representing them accurately. I don’t want to be accused of, oh I don’t know, furiously straw-manning their position by describing an outlandish stance that nobody truly holds and then attacking that instead of accounting for the real problems in my own argument. No indeed, I wouldn’t want to be accused of doing that.

It has been pointed out to me that the bill team are actually deploying a version of the Saatchi Defence in their article, in that they are defending the process they went through when deciding to consult rather than the actual things they did in the consultation. This is somewhat ironic.

Under The Bus

It may surprise you to learn that I actually quite like Earl Howe. Yes, he’s completely out of his depth on the subject of the Medical Innovation Bill, but he’s always been unfailingly polite, and he has always attempted to answer questions (even if the answers themselves have not always been particularly effective at addressing the points raised). He is the only Lord who took the time to write back to me on the subject of the bill, and the addition of a hand-written note thanking me for my views was a nice personal touch. Contrast the Saatchi Bill team, who answer questions only when they are dragged to them, and then misleadingly, as here.

Still, it was an unexpected delight to receive, via the Saatchi Bill team, an unprompted letter from Earl Howe giving further details about how the Department of Health’s consultation was run. Especially delightful, because in the letter, Earl Howe – representing the government and supporting Lord Saatchi’s Medical Innovation Bill – very methodically, carefully, and politely, repeatedly throws Saatchi under the bus. An under the bus blunderbuss, if you will.

The Saatchi Bill team attempt to use this letter in defence of their behaviour, which can be summarised as “A bigger kid told me it was okay.” This article will explain why that explanation answers none of the concerns I raised about the consultation, and in fact highlights a few more problems.

First a quick recap of the consultation timeline, then I’ll go through Earl Howe’s letter. If you feel I’ve overlooked anything, let me know in the comments or via the contact link below.

The Consultation

The timeline for the consultation was roughly as follows (you can skip this bit if you are familiar with the story):

  • The Department of Health (DoH) launches a consultation into the bill, with the methods of response as follows:
    • Via the DoH website.
    • Filling out a questionnaire, then returning it via e-mail or post.
  • The Saatchi team add two additional means of communication via their own campaign:
    • Web form, giving yes/no option.
    • Petition, with a yes option only.
  • These additional communication channels are heavily promoted, in many locations in preference to the DoH methods.
  • 25th April 2014: the consultation closes. The DoH start analysing responses.
  • The Saatchi team start tweeting and publishing articles claiming 18000+ responses. They don’t respond to any questions about how they gathered the data, how it was secured, how they know the detail of the responses when the DoH haven’t published the results.
  • 2nd June 2014, 10th June 2014: The Saatchi team issue two briefing notes*, in which they make various claims for responses to the consultation (the only change in the second version is the addition of further opinion from the team’s legal advisor). They claim 18502 positive responses and only 33 negative, but don’t differentiate between petition and web form responses. The only reference to the DoH methods of consultation response is to say that there were 170 responses, which are not discussed further. There are several pages of quotes in support of the bill, apparently sourced from the Saatchi web form, but no organisation responding to the DoH consultation is named or quoted.
    The Saatchi team do not respond to any questions about how they have arrived at the figures they cite.
  • Lord Saatchi takes the bill through the first two stages in the House of Lords, using the briefing note just described to show claimed support for the bill. In the press, the 18000+ figure continues to be quoted.
  • 30th July 2014: The Department of Health publish their summary report on the consultation. This demonstrates that strong opposition to the bill was received at the consultation stage. Mention of this level of opposition had been absent from Lord Saatchi’s briefing note. No numbers are given for support or opposition.
  • December 2014: Julian Huppert MP makes a request for the numbers and types of responses to the bill. This reveals that the majority of respondents to the DoH consultation opposed the bill, and confirms again that there were 170 respondents.
  • January 2015: A FoIA request by Stop the Saatchi Bill reveals details of the consultation responders. I reported on that in my last piece.
The Letter

Earl Howe’s letter can be found here

Okay, so after the preamble, Earl Howe suggests reading the DoH summary of the consultation. That’s a good start, and it’s exactly what I did. In fact the next couple of paragraphs that Earl Howe quotes were also quoted verbatim in my article. This text makes clear exactly what happened. There were 170 responses to the DoH consultation, and then Lord Saatchi’s team supplied, as part of their own submission, the results of a petition and web form they had run themselves.

But note also what Earl Howe omits here. Lord Saatchi’s briefing note from 10th June, the document that he used to push the bill through the Lords, which was used as the basis and justification for so many articles extolling the overwhelming support for the bill, which was written by Saatchi’s team without waiting for the DoH results: Earl Howe doesn’t mention this at all. In directing us to the DoH document, Howe is confirming that the definitive report about what happened in the consultation is the DoH one, not Saatchi’s, even though Saatchi’s was the one used to describe the consultation to the Lords.

What we appear to have here is another piece of the jigsaw in a key part of my concerns: that Lord Saatchi did not use the consultation to inform his bill, whatever his campaign may subsequently have said. The fairness of the consultation would be totally undermined by this act alone.

The Numbers

Earl Howe next lists face to face events before moving to the subsequent paragraph. And here we have something wonderful.

The purpose of the consultation was not to discover the numbers of those supportive or opposed, it sought instead to reveal the diversity of views on the topic.

I agree with him, it’s not a numbers game. That’s something we’ve been clear on from the start. So here we have Earl Howe (perhaps inadvertently) delivering a mild admonishment to Lord Saatchi, whose campaigning since the consultation has put the numbers front and centre in almost everything they have published, as a key justification for the bill. They used them here, here, here, here, here, here, here, here, here

These numbers have been heaved around by the bill team as if they are very important, in Parliament, to the Welsh Assembly, in the press and on social media, and they are still doing it. In this respect alone it matters what those numbers are. It matters how those numbers were arrived at and it matters what they mean. For example, when Lord Saatchi claims more than 18000 patients responded, how can he possibly know this? His petition did not gather the data to be able to tell him what sort of people responded.

The Petition

Next paragraph:

It was agreed in advance with the Department that views captured through the on-line petition and embeddable web form would be considered.

I think perhaps the Saatchi team have the wrong end of the stick about my concerns. I don’t have a problem with the use of petitions as a data gathering tool. They’re at the weak end of the spectrum in terms of evidence, certainly, but as long as you’re clear about how you went about gathering signatures and the mechanisms you used to ensure accuracy and a fair, informed response, they can still be a useful tool. It’s right that the Department of Health should consider petitions and I would be very disappointed if they were dismissed out of hand. Indeed, I’m sure that if I had contacted the DoH and said that I wanted to submit a petition as part of my response, they would have said fine, go ahead. But it would have been part of my response. The signatures I gathered would have been responses to me, which I then passed on, not individual consultation responses themselves. It would not be fair for me to claim they were. It was the same for Saatchi. This is a significant distinction.

A number of respondents, particularly those responding to the online petition, did not provide detailed responses (often just a yes/no indication of support/opposition). These opinions should not be ignored as they provide an indication of support/opposition to the Bill.

This simply isn’t right. The petition provided no means to register opposition to the bill, it only asked for positive responses. This cartoon by @lexistwit shows the problem.

motion1s

There are many reasons why a petition is weak evidence, but looking solely for positive responses while allowing no nuance or means to register opposing remarks? That is a clear weakness.

Other reasons why you might give less weight to a petition are: if the petition were marketed misleadingly. If the marketing didn’t give people signing the petition an opportunity to see the thing they were signing up for. If the people gathering the data were not impartial. If they failed to control who accessed the petition, or prevent multiple signatures by the same person. If they were not clear about how they were gathering data or processing it. If they failed to respond to questions. If, on top of that, they promoted their petition in a way that could have diverted respondents away from the detailed response method (which would have given a high value response) to their petition (which gave a low value response) without making clear to their audience that was happening. Overall, the weight you attach to somebody’s petition depends on all these factors, on who they are and how transparent they are about their methods. Your attitude to a survey on living conditions for turkeys, for example, may be influenced if you find out it was run by Bernard Matthews, and whether they are prepared to show you their working out.

I leave further consideration of this point to your judgement, since I’ve already made my own views clear. But if only somebody else with a bit more street cred than me could sum this up in a more succinct way, maybe that would be helpful. Ah, here we go. Earl Howe:

Neither, however, can they be treated in the same way as a detailed consultation response from an organisation speaking for thousands of members.

Thank you once again, Earl Howe, for providing another rebuke to Lord Saatchi.

Self-Publishing

So the DoH, rightly, accepted Saatchi’s web pages responses and petition, with the intention of considering them, weighted appropriately in comparison with the responses via the official channels. However, as we’ve seen, Saatchi’s campaign did not attempt to weigh the responses against each other; instead they effectively disregarded all responses from the official channels, ignored the official DoH consultation data and instead favoured the promotion of their own self-collected data, via their own self-published report. In one sense, I suppose, you could argue that the Saatchi campaign did follow Earl Howe’s instruction that one shouldn’t treat petition responses “in the same way as a detailed consultation response from an organisation speaking for thousands of members.” Unfortunately, the way they treated the petition responses differently was to puff them up and publicise them, while effectively discarding all the detailed, considered responses from organisations with relevant expertise and memberships running to hundreds of thousands. As I’ve pointed out before, this doesn’t really help us understand the full range of responses.

Another area where the Saatchi campaign promoted the petition numbers was in their claim that the response to the consultation was overwhelming because the Department of Health had told them 600 responses would be a good level to hope for. I’ve already dealt with this, and shown why it’s not fair to include petition responses, so the actual figure for comparison with the 600 benchmark would be 170, demonstrating that they achieved barely a quarter of the DoH’s target. But it is interesting to see that the Saatchi campaign have, in their defence, cited two other consultations which used petition responses, which allows us to do an easy additional check. We could compare the number of responses to those consultations, to see if the Saatchi Bill petition was indeed overwhelming in its own right as a petition.

Now, it’s brave for Saatchi’s team to cite two examples regarding tobacco marketing, but presumably they decided to stay on familiar ground. The bill team helpfully tell us how many responses were received via petition/campaigning routes to those two consultations. One had more than 96,000 responses, the other had 665,989.

Exactly how overwhelming is 16,500 petition signatures for your bill, again?

Relevance

Earl Howe then goes on to express his gratitude to all those who responded before saying:

The debate on the bill has moved on greatly since the time of the consultation. The current Bill, which will be debated at third reading in the House of Lords has been helpfully amended and improved from the version that was consulted upon.

It seems to me that if there is a discussion to be had about whether the consultation responses remain valid now the bill has been amended, then any doubt that arises will apply to support for the bill, as well as opposition. If the argument is that the consultation is no longer relevant, Lord Saatchi should stop quoting the numbers he claims from it.

Still Bad

Anyway, to finish, I’ll outline each of my outstanding concerns about the consultation, and say whether the Saatchi team’s response has answered them.

  • The Saatchi team, as part of their submission, added additional means of responding (website and petition), and promoted them ahead of the Department of Health methods of consultation, potentially diverting respondents to a lower quality response method.
    Result: Still true. No justification for this has been supplied.
  • The Saatchi team promoted their petition with misleading information, and gave scant opportunity to read the bill itself (none at all on the petition site or annexed pages).
    Result: Still true. No justification for this has been supplied.
  • When the consultation closed, the Saatchi team produced their own report into the results, which did not include analysis of, naming of, or quotes from any of the respondents to the DoH channels. Only positive comments were included. The report was therefore not a fair representation of the consultation.
    Result: Still true. The Saatchi team have remained silent on this, but Earl Howe has added additional confirmation to demonstrate that the official report was not the one Saatchi sent to the Lords. No justification for this has been supplied.
  • The Saatchi team did not respond to questions about how they arrived at the figures they quoted, how they ran their data collection, who had access to the data, or any other response that would have lent transparency to the consultation process or the conclusions they chose to draw from it.
    Result: Still true. This happened. No justification has been supplied.
    The Saatchi team article does give an explanation for why they had said that responses went instantaneously to the DoH while the DoH had said they’d received them in a bundle at the end. The answer: it was both. Assign that explanation the weight you feel appropriate.
  • The Saatchi team have, ever since the close of the consultation, repeatedly claimed 18000+ (sometimes more than 20000) respondents to the consultation, unfairly promoting their petition to a presumed level of evidence that is not supportable. They conflated the results from their web form with those from their petition, which hampered understanding of the results. They then conflated these results with those from the DoH responses, which further confused things. They then made no attempt to differentiate the weight of those results when reporting on them.
    Result: Still true. The DoH has confirmed there were 170 respondents to the consultation. They accepted Saatchi’s petition as a form of evidence for consideration, but the petition responses did not count as individual consultation responses. As Earl Howe says, the responses cannot be treated in the same way as detailed consultation responses from an organisation speaking for thousands of members, but the Saatchi team have done nothing to redress this. Instead, they attempt to bypass this point by answering a different question, the question of whether a petition was permissible at all.
  • The Saatchi team have incorrectly used their numbers to claim that response to the consultation was overwhelming.
    Result: Still true. They have not corrected or clarified this misuse of the figures. No justification for this has been supplied.
  • When the official DoH report into the consultation was published, the Saatchi team ignored it. They did not engage with the majority opposition highlighted in the report.
    Result: Still true. They still haven’t acknowledged the views of the major medical, legal, patient and research groups that oppose the bill. No justification for this has been supplied.
  • The petition, being a petition, did not allow for people to register opposition to the bill.
    Result: Still true. The bill team haven’t answered this point, and Earl Howe mistakenly repeats the misconception. You can’t say no to a petition. How many people didn’t sign it because they couldn’t say no? We have no idea. For a process intended to “reveal the diversity of views, both support and opposition”, it’s a bit of a shame if your chief method of response only attempts to capture the support, not the opposition. This falls far short of ideal, but has never been acknowledged by the Saatchi team.
  • The Department of Health’s own consultation document on the bill was misleading, in that it concentrated chiefly on the overall cost of medical litigation, while making no attempt to find out if any of this cost related to innovation.
    Result: Still true. To date, no example of litigation for innovation has been presented.
  • When Lord Saatchi produced his own personal report on the consultation, the Department of Health should have made clear that their official analysis had not been completed, and that Saatchi’s document was not a fair representation of the consultation.
    Result: Still true. The details are only now just emerging of the strength of opposition to the bill, as a result of FoIA requests. The Department of Health have failed in their duty to ensure the consultation was represented accurately. No justification for this has been supplied.
From Bad to Worse

There’s one additional concern I’d like to raise. This is a new observation regarding the consultation, but it’s perhaps the most important one of all. It arises from this statement in the Saatchi team post:

It was in the clear and mutual interests of the DoH and the MIB teams to work together to gather opinion and was part of the ‘one team’ philosophy developed by the DoH and the MIB team.

Here’s my problem. The government, the Department of Health, as represented by Earl Howe, and the Saatchi team have clearly worked in concert for much of the bill’s progress so far, particularly in the later stages in the House of Lords, where Howe and Saatchi were very much a team. But this closeness muddies the waters when one tries to understand the bill, who represents what view of it and, most importantly, who has what responsibility to represent it accurately and impartially in discussion.

When Lord Saatchi’s team wrote their report for the House of Lords about the consultation process, they clearly didn’t feel a duty to be even-handed or to represent all views. For a stakeholder, a particular interest group, this may not matter. You don’t expect Bernard Matthews to give an unbiased report on turkeys. But Lord Saatchi’s campaign is effectively keeping a foot in both camps, as both an interest group pushing for the bill and, through its “one team” philosophy with the Department of Health, as a part of the body meant to give a fair hearing to all the views, a duty to represent them in the Lords and publicly. What is without doubt is that the consultation process was seriously undermined, and that the Department of Health has been insufficiently rigorous in preventing this damage, all factors combining to contribute to a situation that delayed and seriously misdescribed the outcome of the consultation, while bulldozing on regardless, adopting the misdescription as if it were factual narrative.

At this point, I think my case has been made perfectly adequately in layman’s terms. However, since the Saatchi campaign seem to like to quote impenetrable legal precedent that they think supports their argument, I think it would be useful to hand over at this point for a more professional view from my friend Sofia. Sofia is the most effective possible counter to Saatchi campaign PR – a medic with a law degree – and I heartily recommend her blog. It’s also useful as a lesson for the Saatchi campaign (and perhaps me as well) on how you discuss the law while making it readable and understandable. Over to you, Sofia:

When the government, its departments and any public authority (or one undertaking a public function) run a consultation they are under a duty to do so properly. The DoH, Secretary of State and associated bodies may act in accordance with government consultation principles. They will also remain bound by any relevant statutory duty, a duty of fairness, and a commitment to procedural fairness, transparency and rationality. These are long-established, core and fundamental legal and political principles.

The legitimate expectation exists that a consultation when commenced will be carried out appropriately, with transparency and handled in a procedurally fair manner.   A valid consultation will also abide by a ‘prescription for fairness’ of core criteria and, briefly, the requirement for adequate, properly tested reasons for the proposal consulted upon, timed at a formative stage and with sufficient information presented immediately in the consultation to permit an intelligent consideration and response. The findings or report of a consultation must then be taken into account conscientiously, listened to and paid real attention in any subsequent actions or decisions informed by such consultation.

If a change to proposals has been fundamental, a re-consultation may be required. If their key defence to the negative responses received – that the bill has changed to such extent that the concerns are not relevant – is true, then the proposal as amended should be consulted upon. If it is not true, then real attention should be paid to the concern expressed in the 170 responses by expert and professional organisations, and these concerns should be very conscientiously taken into account.

The 2,000 web responses used a single yes/no question, the 16,000 petition responses only a yes option. Respondents often had no access to the bill on the page of the team’s widget which was spread over the internet including pages of the Telegraph, many of which had wholly inadequate information, if any, about the actual consultation and its factual proposals. The written consultation was worded to steer responses to positive reflection on a matter likely unrelated to the bill’s effects itself. These results are being represented with consequent bias, against all legitimate principles and expectations for transparency in consultation. The negative responses were delayed unacceptably, with the House of Lords and the bill’s team progressing the bill despite the failure to publish or, indeed, consider the consultation findings. Burying expert views obtained through consultation and advancing at such stage is not a transparent, rational, accountable or fair position for the government to adopt. The government are discharging a very important function on an issue that, in altering patients’ current rights, and clinical procedure, stands to affect all patients and all doctors in the country. It is fair to doubt if a consultation can be said to have happened or would stand up to scrutiny.

Thank you, Sofia.

Conclusion

The following statement – made in Lord Saatchi’s briefing note – appears, in the light of what we’ve just seen, to be unsupportable:

Overall, the consultation was effective. It produced a clear and overwhelming majority in favour of the Bill, it highlighted and crystallised concerns and has therefore facilitated a clear and robust redrafting

So will Lord Saatchi’s team correct their previous claims about the consultation with the same prominence as when they were made? Will the Department of Health run the consultation again, in light of a significantly changed bill, and this time ensure it runs properly and is reported accurately? Will Lord Saatchi withdraw his bill until it can be based on claims that are grounded in fact? I assume not. Will Saatchi’s team at least stop making misleading claims about the consultation, and about support for the bill? No I’m not banking on that either.

I’ll leave you with a Ben Goldacre quote.

Science isn’t about authority, or white coats, it’s about following a method. That method is built on core principles: precision and transparency; being clear about your methods […] and drawing a clear line between the results, on the one hand, and your judgement calls about how those results support a hypothesis. Anyone blurring these lines is iffy. Conflict-of-interest stories – where someone has a vested interest in the results of their study – are important, because they tell you when there’s a risk of something wrong in a piece of science.
Ben Goldacre, Guardian 4 November 2011

The Saatchi team have fallen far short of these ideals, not just in their approach to the consultation but in their approach to the entire basis of the Bill. In their rush to drive it through parliament and into law they have trampled on the consultation, cherry-picked results they feel support their cause, and kept their fingers in their ears to considered views opposing them, in the process disregarding the genuine needs of patients, doctors and researchers.

As a result, we have all been let down.

 

Huge thanks to Sofia for her help in preparing this post. Huge thanks also to @lexistwit for the cartoon. Any errors in this article are all my own work. Point them out in the comments or via the contact link below.

* The only difference between the two versions that I can see is that the 10th June edition adds an extra section – Annex C: “Legal Analysis”. As an aside, since no references in the text itself have been altered when the extra section was added, this breaks all the existing references in the text that refer to Annex C onwards.

More Saatchi Bill articles

The Saatchi Bill

widget

Contact me here.

Advertisements

One Response to “Under The Bus”

Trackbacks/Pingbacks

  1. Bil Arloesi Meddygol? Wales votes No | - February 4, 2015

    […] to pass the LCM to agree to the Bill applying in Wales. In considering the Bill Wales took into account its consultation, which though brief received “clear and robust responses”, as did that run by the DH […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: