Research and Evaluation: MylawBC under the microscope

Two documents produced for the Legal Services Society of British Columbia merit international attention. Both relate to evaluations of MyLawBC.com. One is an analysis by external consultants of the website. The other is a specific study by another consultant of outcomes and is intended as a broader  ‘investigation into developing an appropriate benchmark for guided pathway-based websites’. They are important because MyLawBC is a strategic attempt to move beyond the static form of most existing information websites and also because both are serious contributions to the literature on evaluation. To put the position competitively – if you want to be a world leader in the evaluation of access to justice digital provision, these are what you have to beat. Fail to hit this standard and you are second best.

If you are not familiar with MyLawBC and you have the time, it is worth giving the site a quick look. This is the first consultant’s description of its purpose: ‘MyLawBC (http://mylawbc.com) was developed as a user-centric resource and was launched on May 30, 2016. The website is intended to increase access to justice for all British Columbians with a particular focus on low-income individuals. MyLawBC provides information on how to prevent and resolve everyday legal problems by creating customized, action-oriented plans using online tools, actively guiding people to resources, and connecting people to in-person resources via an online platform. Currently, the website focuses on family law, personal planning, wills and estates, and foreclosure issues.’

The best pathway to follow is probably separation. This guides you through questions that log problems and provide you information on responses. It also gives you the kind of helpful background knowledge that a lawyer would tell you eg ‘Legal costs vs. mediation costs A two-day trial costs an average of $39,900. A five-day trial costs an average of $66,850. (Only your lawyer can tell you what the cost of a trial will be in your case.) (Sources: Canadian Lawyer Magazine, “The Going Rate,” June 2015; mediatebc.com.) On average, paid mediation costs $3,044. Family justice counsellors can provide free mediation services.’ Domestic abuse and referral to a family lawyer is flagged throughout. At the end, you get an action plan that includes a summary of your position; more information; a link to a ‘dialogue tool’ that will help with negotiation; and lists of potential referrals. On the dummy facts with which I tested the system (unmarried mother, dependent children, rented accommodation, shared debts), this took 10 pages.

The consultants used a variety of means to collect evidence. These include Google Analytics on use of the website over 16  months; pop up surveys of users offering longer feedback survey to volunteers; ‘key informant interviews’ with users and those involved in the justice system eg as advisers. This seems a pretty good range. Much of the feedback was a bit Delphic. For example, only three quarters of users reported having a legal issue. There were low numbers of repeat users and 90 per cent of all users in a month were new. Rather hearteningly ‘The majority of site visitors reported improved understanding of laws (71 per cent), legal options (65 per cent) and their responsibilities (66 per cent). Around three quarters thought the site useful and ‘most pathway users (71 per cent) reported that they took at least some of the recommended next steps in their customised action plan’. Pathway users got more out of the system than those who just consulted it passively. ‘A minority of both pathway users and non-users (28 and 20 per cent respectively) reported that they had resolved their legal issue since visiting MyLawBC: however, use of the pathway had no significant impact on the likelihood of resolving their legal issue’.

I am no academic expert: far from it. But I take from the above that the system is actually working quite well – with the kind of contradictions that give authenticity. Over the period of analysis, the site had about 3000 users a month. There had been a fall off through 2018 from an early high of 4000 monthly users to 3000 by the end of the year. There seems no clear reason for this. It could presumably be the lack of marketing or the effect of market saturation. 

The evaluation came up with interesting information on users (which, of course, may be skewed by those willing to respond to requests for a survey). It got strong use over the Province – which varies from high urban to high rural. Site users tended to be ‘highly educated’ – near to two-thirds had completed post-secondary education; almost three quarters were women (maybe as a result of family orientation of much of the focus); but ‘personal incomes … tended to be low; one half of respondents had annual incomes of $35,000 or less’ (£21,000 or $US27,000). Website users were overwhelmingly English speakers. Seen in reverse, that suggests that those not using the site had English as a second language and had lower educational levels – suggesting that they may be the more digitally excluded.

The outcomes report had interesting information from Google Analytics. Users coming direct to the site stayed on average 9 minutes – which seems reasonable. The site fared badly on search and was need of some optimisation: ‘For high value search terms, even those copied directly from the site, MyLawBC rarely reaches the first page, let alone the top five’. I would bet that this is a common issue for legal information websites and, indeed, lack of search engine optimisation was precisely highlighted by Kate Fazio of Australia’s JusticeConnect in a recent interview. The authors of the report recommended a sophisticated view of success which should not be limited to completion rate and recommends attention to behaviour, action and attitude. But it is very difficult to measure anything beyond the very basic items of time on the site and downloads. Evaluation can get very subjective once you rely on user’s own assessment. This was a criticism of the University of Twente’s evaluation of the Rechtwijzer.

Together, these two reports raise the bar for evaluation in the field. It would be good if they were taken as setting a standard. Indeed, as a bare minimum, revelation of the Google Analytics on websites would be useful. Projects are often pretty cagey about the actual use that they are getting and that is hiding issues like the need for search engine optimization. More generally, it is interesting to turn to the list of questions raised in 2015 in a report for the Legal Education Foundation on the Rechtwijzer evaluation. We continue to need some serious study of the answers:

‘At some stage, the Rechtwijzer [you could add MyLawBC] is going to have to be tested against the criteria of the more sceptical in its audience notably:

1. What proportion of the population (particularly that historically eligible for publicly funded legal aid) can be shown to derive benefit from website-based provision (ie how many are excluded by reasons of lack of access and adequate skills such as literacy?

2. Does Rechtwijzer 1.0 [MyLawBC] meet its objective of adequately signposting users to effective (and preferably conflict reducing) solutions to their problems?

3. Does it do this better than traditional individualised assistance?

4. Does it adequately ‘red flag’ out those cases (notably involving domestic violence in the family) for which legal representation is available (in most jurisdictions) and desirable (in all)?

5. To what extent does website-based provision depend for its greatest effect on integration with some measure of individualised assistance and what form of such assistance works best and provides most value for its cost?

6. How good is the advice and information provided and, in particular, does the Rechtwijzer [MyLawBC] adequately protect the rights of weaker parties – traditionally women in family disputes?

7. … [specific to Rechtwijzer]

8. For those … who assert the Rechtwijzer’s superiority over traditional non-interactive websites, is there any empirical evidence to sustain this argument?

Intuitively, MyLawBC looks to me like a sight of the next generation of websites that make the existing information sites like citizensadvice.org.uk two dimensional an old-fashioned. It would be good to have some evidence to back that up.

Leave a Reply