synapseproject.org

Dear John... oh how I hate to write
(hmm... maybe that's the crux of a PBRF-related issue right there)
Dear John, I must let you know tonight
That my love for teaching is gone
So I'm sending you this song
Tonight, I'm with another
You'd like him John, he's got a quality score over 6.5
So I'm sending you this letter
Dear John.

 
The research productivity of New Zealand universities is evaluated on a six yearly cycle, and this evaluation defines a substantial part of the research funding they will receive for the next six years from the Performance Based Research Fund (PBRF). Unlike most overseas equivalents, in New Zealand the unit of analysis is the individual academic, not the department.

As a Wellington-based academic who is not at Victoria University of Wellington, I've experienced a different side to the local news coverage of the PBRF results. What a strange beast PBRF is. Firstly, how great it is that The University of Auckland and Otago University (the only two New Zealand institutions with medical schools) weren't topping a list for once. And trust me that we have all heard that Victoria was the "top ranked" university. The average "quality" scores were, of course, the only metric that gets any attention. Did you know, however, that despite being "ranked sixth" on our "quality score", my current employer Massey University will actually receive 28% more PBRF funds than Victoria in 2013, based on the outcomes of the round? And that in the "quality evaluation" component specifically, Massey will receive 37% more funds than Victoria? How is that? Surely since the PBRF is about distributing funds to where they are deserved, they would be going to Victoria? No. PBRF funds are based on the total amount of good quality research being done by an institution, not just how good it is on average when divided by the number of portfolios submitted. It's like comparing a bespoke dining table that has been lovingly polished till it gleams, to another home with an entire good quality dining suite with chairs plus also a bedroom suite. If you do less, you might indeed make it better on average... but you've still done less in total. Apples and guavas.

Here's another angle I see on this. Did you also know that I would lower the quality score of any academic psychology department in the country that I joined? What a great feeling. This, despite having been the New Zealand principal investigator of a four year, three site international clinical trial, with colleagues in the US and Canada, funded by the US National Institute on Disability and Rehabilitation Research during the PBRF period? This, having supervised four doctorates to completion during the period, another since, and with five further doctoral candidates submitting for examination in the first half of 2013? And also coordinating Massey's Doctor of Clinical Psychology programme on our Wellington campus, with over 20 concurrent doctoral candidates throughout the past three years? And in addition to my other teaching, having also taught literally hundreds of frontline brain injury rehabilitation professionals, service managers and case managers through professional development courses during the PBRF period, meaning there may not be a brain injury rehabilitation service in the country that doesn't have staff that I have personally taught? Opportunities for genuinely health service-impacting research abound.

As it has been structured in New Zealand, the PBRF is fundamentally an exercise in individual behaviour change at a mass population level, an issue that I have a strong personal and professional interest in. Despite this, to my knowledge psychologists haven't been involved in the design or evaluation of the PBRF. In my opinion, the behaviours that have been observed at an institutional and individual academic level in the PBRF process were entirely predictable, and in many cases, unfortunate.

I'm moving to AUT University in Auckland in July. I'll be working for the next year as a Senior Lecturer in Clinical Rehabilitation in the School of Rehabilitation and Occupation Studies, in a position where I'll work with the Person Centred Research Centre. I'll be spending about half my time in translational brain injury rehabilitation research in partnership with ABI Rehabilitation (who are mostly funding the position) and other frontline brain injury rehabilitation services in Auckland and to some degree throughout New Zealand. The other half of my time I'll be back at AUT's north shore campus, primarily pursuing other neurorehabilitation research. I might be involved in supervising one or two graduate students, and I'll give a few guest lectures during the year possibly. But the primary focus will be research, working with a great team who are both highly productive and highly personable. I enjoy teaching (when I have the time to do it properly) and my teaching evaluations have frankly been far better than the evaluation I received from the Tertiary Education Commission through the PBRF process last week. But I am more passionate about impacting brain injury rehabilitation services in New Zealand and internationally through relevant, cutting edge research. So you could say that in my new job I've won second division PBRF Lotto. (First division would have been to get a Rutherford Discovery Fellowship.) I've found an exit that will resolve the workload strain that has been affecting my family life, while also positioning me for future research productivity. But most of my academic colleagues here have not seen greater space being created for them to do more research or better research through the PBRF process... just additional pressure to be more productive alongside all their existing commitments.

I have personal knowledge of, and deep respect for, some of the people who designed the PBRF. And I'm sure the PBRF was designed for excellent reasons. But something needs to change here. Because our university system isn't just broken—it's breaking people.

A holiday season is here. For those in the southern hemisphere like myself, the Christmas period marks not just that holiday but the conjoint start of the summer 'vacation' period (we don't call it that down here), which in northern climes is much more sensibly segregated to the other half of the year. But regardless of whether your Christmas may be white or sunburnt red, it is a time when many of us will be taking at least a few days away from the working environment. As I've been heading into this period, I started to think about the experience of people with disabilities in this holiday season, and particularly of those who are resident in rehabilitation services. We know that many people—like Dan—want to get home for Christmas and other holidays. It looks like Dan will, but many people won't be able to do this.

I set out therefore to see what the peer-reviewed literature could tell us on this topic. My attempts to find accumulated knowledge about people's experience of a holiday season spent in rehabilitation have been fairly unsuccessful, however. The only literature I could find on holiday-related environmental manipulations in health services was recent (highly quasi-) empirical evidence that indicated tinsel is harmful not just to pets but also to blood gas analyzers. The authors light-heartedly suggest Christmas decorations could be an impediment to patient care (or at least infrared touch screens, anyway). Yet we do know that small environmental differences can have important psychological implications. It was demonstrated in the 1970s that older adults given care of potted plants have a mortality advantage over their peers who have possession of a potted plant but aren't charged with its care—yes, we humans are less likely to die if a potted plant needs us. So tinsel would seem worth the risk. What else should we do beyond this, however, to make our rehabilitation services places that can be a positive place to spend the holidays? And what is the effect for clients when a highly valued holiday season fails to live up to previously cherished beliefs about the way it is 'supposed' to be?

No doubt all inpatient rehabilitation services make efforts towards a more festive environment during the holidays. I confess I have only a vague recall of the good efforts other staff made (probably mostly our nursing staff) when I worked as a Clinical Neuropsychologist at the Wolfson Neurorehabilitation Centre in London in the early naughties. To my shame I wasn't ever involved in festivities at the Wolfson on the holy day itself. My lack of anecdotes thus mirrors the paucity of published information on this topic—apparently, a complete absence? So it'd be good to hear some of your experiences. If you've got a heart warming, sobering, or enlightening story of the experience of neurodisability and the holiday season in rehabilitation services I'd like to hear it. Please email me if you're able to share not just with me but with others—being appropriately mindful of confidentiality. I'll distill what I can, and share thoughts back in an update in the next week or two to our community here. And as we reflect upon these issues, perhaps this can be the start to a deeper appreciation of how to provide more supportive holiday seasons in future years for any clients who have felt somewhat unseasonal in our services in the past.

Wishing you and yours a safe and peaceful holiday season.

"Common sense is not so common."—Voltaire (1764)

I was reminded of this widely re-quoted saying when reading Instilling a research culture in an applied clinical setting, recently published in Archives of Physical Medicine and Rehabilitation. In their paper, Dr Michael Jones and colleagues are clear and thorough in outlining a wide range of practical issues and considerations that arise in pursuing the goal of integrating research into our clinical services.

At one level it could be easy to read this paper lightly, and risk dismissing their suggestions as 'just common sense'. This would be a mistake. There are at least two reasons for this. Firstly, there is the comprehensiveness of their coverage of the many considerations that may arise. I think few clinical settings would have fully worked through implementation of every suggestion in this paper—so there are practical action points we can all take away from this paper. Secondly, one of the most helpful aspects of genuinely good advice is the way it doesn't recommend some alternatives that might also sound good on first hearing, but could ultimately lead to undesirable outcomes. I'm not claiming here to have a deep grasp on all of the things we shouldn't be doing, but my reading of this paper was that it offers good advice—both in what it does say, and what it does not.

While a few aspects of the paper speak specifically to the United States context (e.g., information about funding agencies), these should not detract for an international audience. And if you're a clinician or manager who wants to begin the process of bringing a research culture into your organization, this paper will provide a dozen ideas for where you could start.

The Lancet Neurology last week published Incidence of traumatic brain injury in New Zealand: a population-based study. Prof. Valery Feigin from AUT University, with Dr. Alice Theadom, Dr. Suzanne Barker-Collo, Dr. Nicola Starkey, Prof. Kathryn McPherson, and colleagues, reported on their impressive study that applied a fine sieve to an entire urban and rural catchment population of over 170,000 in the Waikato region of New Zealand for a one year period. This is the most thorough incidence study of traumatic brain injury that has been conducted—the first large scale population-based study covering both urban and rural areas. The study defines a new standard for future research in this area. (It metaphorically but also literally defines a standard—see the paper's Panel 2: Suggested criteria for population-based studies of traumatic brain injury incidence and outcomes, p. 10.)

The case identification methodology is impressive:
”We aimed to assure complete case ascertainment using multiple overlapping sources of information about all cases, admitted and not admitted to hospital, both fatal and non-fatal. This case ascertainment included the following: daily checks of all public hospitals and emergency departments (including surgery and neurosurgery departments) in the study region; monthly checks of CT and MRI records, hospital discharge registers for public and private hospitals in the wider Waikato region, family doctors, rehabilitation centres, and outpatient clinics; quarterly checks of coroner and autopsy records and rest homes; and a yearly check of ambulance services, the prison located within the study region, and the Accident Compensation Corporation (ACC) database. The ACC is a government-supported no- fault insurance agency that funds treatment and rehabilitation for all New Zealand residents with injuries. Cases were also identified from the national death register (we ascertained all death certificates with any mention of TBI). We made every effort to capture data for all individuals with mild TBI who were not admitted to hospital, by including those from family doctor practices providing direct referrals of new and suspected cases of TBI, and by doing checks of accident records of community health services, schools, and sport centres (within and just outside the catchment area), and through self-referrals (the study was widely advertised in the study area via television, newspaper articles, and newsletters and posters). Final checks for complete case ascertainment included reviewing computerised hospital separations data (deaths, discharge, and transfers) for public hospitals with ICD-10 S00-S09 codes for head injury (via the National Health Index number). All TBI cases were checked against existing cases in our TBI registry, to identify any duplicates. Remaining suspected cases (ie, cases for which the presence of TBI was not clear and needed to be verified) of TBI were cross checked with hospital discharge lists, hospital inpatient management records, lists of excluded cases (ie, TBI criteria not met, individuals who did not live in the study area at the time of injury), and lists from other sources (ie, schools, sports groups, rest homes).” (pp. 5-6).

By now it will not surprise you that their evaluation of case information was equally as thorough once participants were identified.

The study identified 1,369 traumatic brain injuries that occurred during the study year, including 71 moderate to severe injuries. This equates to an overall incidence of 790 cases per 100,000 person years. The authors note this was substantially higher than the incidence observed in other high income countries in Europe (47–453 cases), North America (51–618 cases) and is also higher than World Health Organisation estimates. It is possible that there is something different about New Zealand. However, given the rigorous methodology of this study the more likely outcome is that future research will confirm this incidence rate in other high income countries as well. The authors note that regrettably even higher incidence rates again are expected in lower income countries.

There are many ways in which good epidemiological data contributes to health service delivery. Good data guides injury prevention efforts. The kind of partnerships this study describes is an example to us all. Building and maintaining such networks for not just research but clinical purposes is a goal worthy of consideration in itself. Meanwhile, a key question that arises for rehabilitation services is: what is the outcome for the many, many mild (and moderate) injuries that are not being captured into the health system whatsoever, let alone receiving rehabilitation services? Do they spontaneously make a good recovery? How much worse off are they than those who receive services? Given the high numbers of people not accessing services, are there additional population-based interventions we could provide to mitigate at a distance some negative outcomes? On the whole, we don't have good answers to these questions. With this new high water mark set for how many injuries are occurring, the importance of these questions is further underscored.

The most important skill in rehabilitation is evaluating the outcome of interventions with each individual. The most important knowledge in rehabilitation is how to systematically conduct such evaluations using a single case experimental design.

This may not be self-evident. No doubt there would be dissenters to these statements. And clearly, there are many other skills and much other knowledge that we would expect in any competent rehabilitation practitioner. For instance, the ability to skillfully assess capacity and impairment, and contextualize these in relation to community integration, is important. An awareness of evidence-based approaches to rehabilitation, and the ability to effectively bring knowledge from their particular professional background to bear on implementing these, is likewise important. Yet, as a field we're now well aware the limitations of randomized trials—in particular, that effects observed on a group basis in carefully selected sub-populations can only get us so far in knowing what will work for a specific person at a specific place and time. So while clinicians consider the available evidence, draw on knowledge and experience, and provide the best interventions we can, we need something more.

As an undergraduate psychology student I completed a course on psychophysics. In one laboratory session in an old house off the outskirts of campus, the professor demonstrated the next task. Placing a pair of heavy, sound-isolating headphones on, he spoke into a microphone. His voice was returned through the headphones and he spoke normally. A switch was then thrown, introducing a delay of a couple of seconds into the audio feedback he was receiving. Coherence dissolved. His speech became slurred, halting, irregular—barely interpretable at times. Removing the headphones, he explained that fluent human speech is predicated on the continuous feedback mechanism inherent in listening to our own voice. When this is disrupted, we are thrown off track. He noted that some are more effected by this than others, and that his wiring left him particularly susceptible. We were invited to experience derailment.

What matters most in rehabilitation is long term outcomes, yet in many cases clinicians receive only scant information about whether their interventions continued to work in the short term beyond discharge, let alone over the years beyond. In essence, in clinical practice our feedback loop often appears to be disrupted. Goal setting and outcome evaluation are now appropriately commonplace in rehabilitation services. However, pre- and post-treatment assessments in themselves are insufficient to systematically demonstrate that an intervention was the cause of any changes observed for the person receiving rehabilitation. We know that there are factors such as spontaneous recovery that are directly correlated with time, thus time spent in our rehabilitation services, and thus progress through treatment. To close the feedback loop, we need a way to rigorously demonstrate whether an intervention worked, with this specific person. Fortunately, that methodology already exists: the single case experimental design.

Work in this area is being pursued most vigorously by Prof. Robyn Tate at the University of Sydney and various colleagues in Australia. They have been iterating a measure to distinguish high quality, well controlled single case research designs from qualitative case studies and steps in between. Initially published in 2008 as the Single Case Experimental Design (SCED) scale (available for download from psychbite.com), while a revised version renamed the Risk of Bias in N-of-1 Trials (RoBiN-T) is in development, and adds further sophistication. An evolution of the PEDro-P scale for evaluating Randomized and Non-Randomized Controlled Trials, these scales were designed for evaluating the quality of N-of-1 trials. Beyond this, these scales provide an overview of the considerations in designing a good single case study from the outset. As a result, this work is compulsory reading for all my postgraduate students.

Implementing single case study experimental designs in clinical practice will not necessarily be straightforward. The Neuropsychological Rehabilitation SIG of the WFNR had their annual conference in Bergen, Norway in July 2012. Among much good work presented there, Dr Henk Eilander of the Department of Brain Injury, Huize Padua, Netherlands, presented their poster, Feasibility of single-case study designs to evaluate neuropsychiatric treatment of behaviour disorders after severe acquired brain injury. (Dr Eilander was good enough to allow the poster to be hosted on this site so you could access the full content of the poster.) A key conclusion: "Although case studies are feasible in a clinical setting with limited resources, the naturalistic character of this study as well as the inexperience with systematic research resulted in too much variability to be able to draw firm conclusions on the effects of the medication." My reflection from their valuable study—we clearly need to be devoting (even diverting) resources to develop these evaluation skills in our frontline clinical services.

Clinicians need powerful, simple-to-use tools so that it becomes an obvious choice to apply single case experimental design methodology in routine practice. At that same July conference, Robyn Tate, Dr. Michael Perdices and colleagues demonstrated their current work to develop an online training program to guide and accredit RoBiN-T raters. Through this online tool trainees are coached to correctly evaluate and rate single case experimental design studies, and compare their ratings to an expert consensus panel. While primarily designed to train people to be raters, in undertaking this training they learn the core skills to undertake such a study themselves. When available, I think this interactive tool will immediately become the best starting point to develop a grounding in single case experimental design.

Neuroscience is providing us with remarkable data on brain functioning. A reductionist dilemma. Everything from chemistry to sociology could be explained in terms of physics, at least theoretically. Yet, "I ran the car into the wood chopping block" is a far more evocative story than any precise description of the physics of the event. While supplying a complex mathematical equation might most precisely define the effect on the car, it is the story that enables my wife to picture the damage I have caused. Ah, but what if the story was, "I think I ran the car into the wood chopping block"? And (humor me here) what if you couldn't go and look at the car to check? A description of the forces on the car would help separate a liaison with a former tree from the bump of just running over a pothole.

Dr. Keith Cicerone presented the Facts, Theories, Values: Shaping the course of neurorehabilitation. The 60th John Stanley Coulter Memorial Lecture. Many valuable points in that lecture you'll need to read yourself. He observed, however, our growing understanding of neuroplasticity, informed primarily by advances in neuroscience:
The roles of behavioral variability and predictability are central to recent investigations of executive functioning in relation to the frontal lobes. In this framework, executive functioning (task setting and monitoring) is related to the dorsolateral frontal cortex; emotional and behavioral regulation is related to the medial and lateral orbitofrontal cortex; and the rostral prefrontal cortex is related to "metacognition" involving the integration of motivational, emotional and cognitive activities. These same processes—involving the supervisory attentional system and anticipatory neural network—are central to the rehabilitation of cognitive impairments through meta-cognitive strategy training, characterized by interventions to foster anticipation and planning, response monitoring, and self-evaluation. This may represent at least the beginning of a theory of cognitive remediation that integrates neurologic, neuropsychological, and rehabilitation concepts and mechanisms…

In our own work, we have suggested that meta-cognitive strategy training directed at improving patients' self-regulation of both cognitive and emotional processes leads to increases in patients' self-efficacy beliefs, specifically in their confidence in managing residual cognitive and emotional symptoms. Improvements in perceived self-efficacy (and related concepts, such as maintaining a positive problem orientation) are directly related to positive outcomes, particularly patients' subjective well-being and life satisfaction.

This approach to rehabilitation puts the patient's subjective experience and beliefs at the center of the rehabilitation process.

Returning to physicists vs. chopping blocks, our neurorehabilitation practice often seems most like the uncertain story—"I think… perhaps it could be..." Our formulations of the difficulties facing the people we work with are informed by many things. We draw on theory, on observation, on formal assessments, on information about the impairment to the brain, and hopefully by listening to the person's own experiences and perceptions, as well as those of the people around them. We make our best attempts to draw a connection between these explanations and our knowledge of what works in neurorehabilitation interventions. And then we do whatever it takes. And that's not so bad.

A clearer neurobiological model of brain processes after impairment, and how they change in rehabilitation, could plausibly turn much of this on its head. We might discover some chopping blocks were potholes, and some potholes were something else altogether. But a clinician can't solely be a physicist (or a neuroscientist). Higher level subjective human experience is imbued with meaning not circumscribed by neurobiology. It seems to me that in order to apply advanced neuroscience insights into neurorehabilitation practice, we're going to need a much clearer map to guide us.

In a recent paper in Neuropsychological Rehabilitation, PDA and smartphone use by individuals with moderate-to-severe memory impairment: Application of a theory-driven training programme, Svoboda, Richards, Leach and Mertens describe the provision of mobile computing technology to ten people with acquired brain injury. Their participants engaged well with the technology and a rigorous ABAB design provided strong evidence that the intervention was effective for these participants. In somewhat of a departure from usual practice, this paper does not state the smartphone or PDA models that were used by the ten participants. The authors do reference individual case studies they had previously published about two of their participants. One of those case studies described the use of a Treo 680, while the more recently published case study did not report the actual technology used. In the absence of other information it seems likely that first generation smartphones like the Treo 680 were the newest devices used in the trial, and references to PDAs indicates some older Palm Pilot devices were also likely used. Perhaps device models were not described as the authors saw this as a potential distraction from the results, which they felt remain applicable to more recent technology.

I agree that findings based on earlier devices should certainly still be considered relevant, a point I make in a chapter in the forthcoming Oxford Handbook of Clinical Geropsychology, and which was also discussed by Gillespie, Best and O'Neill (2012). However, it remains possible that future research will demonstrate important differences in the effectiveness of one generation of technology over another. It is a pity that a clear description of the technology was not provided by Svoboda et al.—it was, after all, integral to the intervention. Even if reporting somewhat older technology, they remain in good company. At talks given to both the APS College of Clinical Neuropsychologists in Brisbane and to the University of Queensland School of Psychology last week I noted that as far as I am aware, no study to date has been published in neurorehabilitation that describes the use of modern touchscreen smartphones. This is despite it now being over five years since the Apple iPhone redefined this market in 2007. While studies with touchscreen devices are no doubt coming, this starkly reflects our disconnect: compared to the rate of evolution in technology, academic research is glacial.

The authors placed an emphasis in this paper on the training approach they used. Svodoba et al. used relatively intensive clinician input—an average of eight hours of individual training with the device per participant—and an errorless fading-of-cues protocol to train their participants to use their devices. (This is similar to the approach described in the Australian study that was the focus of the most recent Synapse Voices podcast with Belinda Carr and Natasha Lannin.) The training approach appeared well-considered and suitably based on theory and past research. It is worth reflecting that the authors did not actually test their belief that this rigorous training approach was necessary to generate the positive changes observed in their participants. The methodology used in the paper is only capable of demonstrating that such an intervention was sufficient to generate the change. It is not impossible that a less intensive training approach may have been equally effective, something it would be useful to examine in a future study.

Svodoba et al. are to be commended for using mainstream, shipping technology in their research. More than two decades of research into customized mobile computing devices as cognitive prosthetics has resulted in precious few devices or services that can be used in further studies by other research teams, let alone products that can actually be issued to clients to assist with their day-to-day difficulties. Steve Jobs, founder of Apple Inc., is reputed to have said, "Real artists ship". Though famed for a perfectionist attention to product design and interface detail, Steve Jobs understood something that too few people do—that value is only generated when a product is actually shipped for widespread use. By focussing their research on mainstream technology and services, Svodoba et al. are setting an example that all of us should follow. Rehabilitation needs actual shipping products.

Dijkers, Murphy and Krellman, Evidence-Based Practice for Rehabilitation Professionals: Concepts and Controversies, was released online in the Archives of Physical Medicine and Rehabilitation last week. They trace four decades of change in what is regarded as evidence, and two decades of our field attempting to employ evidence based practice. The authors review the various criticisms of evidence based practice, demonstrating they largely hold little validity and should not be barriers to applying evidence to our practice. They note, however:
"...knowledge of the principles and procedures of EBP alone may not be enough to actually use research evidence in practice. McCluskey and Lovarini tested the effectiveness of a 2-day interactive workshop in changing knowledge of and attitudes toward EBP, and tracked patterns of implementing EBP (eg, searching literature, appraising research, and using it in practice) for 8 months after the workshop. Although knowledge and attitudes significantly improved over time, there was very little implementation."

Innovation guru Tim Kastelle this week described handwashing in hospitals as the most important innovation ever:
"Oliver Wendell Holmes in the 1840s was one of the first people to suggest that hand-washing could reduce infections. Not many people paid attention.... There was about a seventy year gap between Semmelweis proving that hand washing saves lives until the practice was widely accepted. Even today, in many hospitals less than half of the health care practitioners follow the right procedures for hand washing."

Dijkers, Murphy and Krellman's paper is worth the attention of every reader. It is likely we all have some less than rigorous rehabilitation practices that we should wash our hands of. In that vein, Dijkers et al. offer specific, practical strategies that may assist. But if the experience of hand washing tells us anything, it is that mere knowledge of an effective intervention—even intimate knowledge—and the clear capacity to implement the intervention, are not alone sufficient to generate lasting, consistent behavior change.

At the time of original posting, this article was not accessible via its DOI. Until that was corrected, this update originally used this alternative link instead...