National Post (National Edition)
Bad research basis for law debate
Lawyers. Is there any other profession with such an exaggerated sense of its own value and significance? Whenever I hear something new about the statement of principles that the Law Society of Ontario (LSO) is imposing on all legal professionals in the province — mandating lawyers and paralegals to draft a statement about how much they personally value inclusivity, diversity and equality — I think of a joke I once read. Question: What’s the difference between a lawyer and God? Answer: God doesn’t think he’s a lawyer.
OK, the joke is not — technically — phrased in a logical way, but it’s still funny because, like the LSO, the legal profession does seem to think of itself as being able to do and understand all things. And the reason I felt compelled to mention that the joke is technically misstated, even though this is unimportant to my overall point, is because I’m trained as a lawyer.
Of course, the reality is that lawyers are flawed human beings, the latest evidence coming in an expert report prepared by Prof. Rod Clifton for the Canadian Constitution Foundation (CCF) in its constitutional challenge of the LSO statement of principles, which CCF considers forced speech.
As a lawyer, I will disclose here that I am a former executive director of the CCF, so infer any bias you care to. As a writer, I will at last get to the point: Dr. Clifton found that the foundational research on which the LSO based the mandatory statement of principles is extremely flawed.
That is a significant piece of news because without its finding that there is systemic racism in the legal profession in Ontario, the LSO would not have instituted the compelled speech requirement.
So, what do I mean that research on which LSO relied is extremely flawed? Dr. Clifton — a sociologist who has taught social research methodology courses for almost 40 years — offers around 20 pages detailing the serious problems. (You can read his full report on the CCF website.)
The condensed, columnlength version is that there are four main deficiencies in the study — which was designed and conducted by Strategic Communications Inc. at the request of the LSO’S Working Group on Challenges Faced by Racial- ized Licensees and the LSO’S Equity Initiative Department.
(Incidentally, I’m sorry that the LSO recently changed its name from the Law Society of Upper Canada to the Law Society of Ontario because the shorthand “LSO” brings to mind a big classical orchestra, while “LSUC” somehow conjured a more accurate picture.)
It turns out the first fault marring the research is the sort of error one learns about in Research Methodology 101: confirmation bias. Focus group participants, survey respondents, and key “informants” (people whose answers were used to reach the study’s ultimate conclusions) were presented with questions that made repeated statements about the effects of racialization on the success of lawyers and paralegals.
Being exposed to these statements so many times made the study participants more likely to confirm the statements. The repetitive construction of the ques- tions unduly shaped the responses.
That’s likely of lesser concern than the second problem, which is that in all these surveys and conversations and focus groups, it’s entirely possible that no one knew what they were talking about. Or at the very least, no one trying to interpret the results can be sure of what the respondents were talking about. The term “racialized” is new, confusing and imprecise, and not even the authors of the study could decide what it meant, telling survey respondents that the term “is either or both an imposed or chosen self-identity” (respondent’s choice!).
That means that it’s not even possible to know what the study was measuring. In effect, any lawyer or paralegal of any colour or origin could be accurately classified as “racialized” … and also as “non-racialized”; it just depends how that lawyer or paralegal feels.
But there is a special place in research methodology hell for the third issue with the report: the sample was not representative of the population about which conclusions were made. To have obtained meaningful results about Ontario lawyers and paralegals, it would have been necessary to hear from some sort of random sample of these people. Instead, the study based its survey conclusions on the responses of 3,237 self-selected lawyers and paralegals. The rest of the over 40,000 chose not to participate.
The fourth identified problem with the study is the absence of control variables, which makes it impossible to accurately assess cause and effect. Other than that, truly great work, LSO.