Judging judges
The current pipes|drums Poll indicates that almost 90 per cent of the world’s pipers and drummers feel that “associations should have a system for competitors to provide feedback on judges.” With such overwhelming desire for competitors to judge judges, you have to wonder why it doesn’t happen more often . . . or at all.
The only attempt I know of to tap competitors for their opinions on the merits and abilities of adjudicators was when the Competing Pipers Association did a survey of its members maybe 10 years ago now. It was done by traditional post, and respondents were asked to grade a list of maybe 75 pipers who had judged events. From that, the CPA was able to work with the new Joint Committee for Judging, and weed out not a few obvious people who clearly did not have the respect of those they judged. And, as we all know, if a judge isn’t respected, the result isn’t worth, as Seumas MacNeill once said, a proverbial pail of, um, spit.
So what’s stopping the world’s piping and drumming associations from asking their members for feedback? I can’t think of any good reasons but the familiar matter of time, since such a program would take concerted effort not just to execute, but then act upon.
I suppose also that not a few judges out there may feel a little threatened by such an initiative. Every piping and drumming judge – at least outside of the UK – was once a competitor, so he/she has experienced the frustration of receiving an ambiguous or even insulting score sheet, or an adjudicator with poor decorum, or the stinky air of blatant conflict-of-interest. Personally I would be very suspicious of any judge who loudly protested a well constructed feedback system.
And by “well constructed,” I mean a system that assures that competitors can respond in confidence, assured that their identity is never revealed but that their opinions are considered equally.
By hearing the compliments and complaints of competitors, I would think that each organization could then learn and work to improve conditions. Judges can learn to be better. The results become more respected and credible. Bring it on.
Unfortunately I’m sure there are some associations where there are so few judges to choose from, that the incumbents may not feel any pressure to change or improve.
The same could be said for associations that have a large judging pool but far too often select certain personalities. I’m not confidant enough to believe or accept that anyone submitting a statement to an association re a judges performance, would in fact remain anonymous. Isn’t it the responsibility of the association and it’s executive to make sure that whoever is on the panel is worthy of being selected. Judging for some can at times be a futile experience whereby everyone i.e. the competitors can’t always be joyous or even satisfied and far to often are honest or realistic re their performance. Wouldn’t this approach open up the gates to possible innuendo and bias towards judges, coming across as negative rather than constructive. Granted there may well be a few judges who fit the bill but what about the many good judges who may end up getting the raw end of the stick? Perhaps the associations and it’s executives are best equipped to monitor this criteria?
Correction To Previous: are NOT honest or realistic re their performance
In New Zealand every year association bands elect the judges from a list who will be judging at the national contest. The only exception to this is that every year one or two overseas judges are invited and they are not subject to an election.
BC Pipers surveyed its competitors a few years ago. The survey addressed the overall operations of the association but had several items specific to adjudication.
I think it is up to the respective associations to hold their judges accountable for their performance. Regular continuing education, re-testing every so many years, review of score sheet comments, etc. Frankly, the individual player is too biased. That;s what we pay dues for, to delegate to the associations decisions we are not capable of making ourselves. But no matter where you stand on this, the standards for judging need to remain high. I know of a judge in a U.S. association who lied on his application. He never actually competed with the highest profile bands he has on his resume. His references were checked, but somehow he is still a judge depsite only having competed in two grade II band contests in his life and nothing higher. How does that happen?
No matter how well-meaning I know this idea to be, I seriously worry about a system where judges are “judged” by competitors. Even after some sort of criteria was established I wonder how input might come close to be fairly vetted, or, “judged” for fairness? For example, are those offering input (judgement) of the judge in a place to provide considered feedback? Do they have credentials, experience and maturity that add up to fair judgement, fair judgement of the judge? I’m no legal expert only an average citizen of a good democracy. I know judging the judiciary is a tricky and not all-that-common business, by citizens, especially. While citizens (our courts of law equivalent of competitors?) may rate judges in some informal sort of ad hoc way, citizens don’t generally judge our judges. It strikes me that superior courts judge the judgements, the performance, of a judge, with the Supreme Court having the last word. I’d hate to think a judge in a court room, one deciding my fate, may be there because he managed to offend the least number of the citizenry and be a wish-washy dope [recognizing that in some countries lower court judges are often elected by the general voting pop].
Anyway, I’m not a fan (be very suspicious!). I am a bagpipe/band judge because I passed a gate of credentials followed by a series of examinations and tests. I abide by my association’s Code of Conduct. Like any judge, I can me censured at any time for sub-par results or behaviour. I suggest that your referenced case of the CPA’s rating of judges was related to an extraordinary time when governing organizations were not doing their job. That CPA assessment was a sort of glorious mutiny. Anyway, it seems to me just rules and strong, fair-minded leadership will address any issue that arises related to poor judges with poor judgement. I can think of some our best judges (in my own humble view) and , while they consistently deliver fair results, can imagine them low-down in an adjudicator’s popularity contest. That doesn’t seem fair.
Good points, MG. The difference between citizens judging the judiciary and pipers and drummers providing feedback is of course that piping and drumming competitors are assessed by their association’s judges regularly – in some cases (e.g., yours) hundreds of times. The players can become very familiar with judges’ strengths and weaknesses. I’m not suggesting at all that judges would be summarily removed, or, for that matter, given a medal, only that if trends are identified associations might be able to improve their own “judiciary” by working with it to do an even better job for the players.
What would happen elsewhere is that people are trained, then there’s a probationary period with some close supervision, then thereafter within the organisation there would be a structure in place to evaluate in an on-going way, the standards and consistencies of the judges/examiners, whatever their level of experience. There would also be a requirement for ‘x’ number of hours of Continuing Professional Development every year in order to stay on board. Within the marking system of The Associated Board of the Royal Schools of Music, for instance, ‘judges’ (examiners) are consistent to within 2 marks of each other, such is the degree of standardisation of marking that has been worked out by the Association, and each and every examiner is thoroughly trained to adhere to that. But that’s where there are set tunes and set requirements. Maybe it’s a lot harder with ‘free-style’ events such as Medleys, and maybe there’s a difference between an examination system and a judging/competition one. Is there some hope, that with the newly trained judges coming through the systems(s), they will be more ‘with-it’, more enlightened, fresher, and bring a less narrow-minded stance to the whole business of judging?. The hope would surely be that a new breed of judge would WELCOME feedback on their performance, would WANT on-going development opportunities, would CONSTANTLY be evaluating their own abilities and competencies.
Janette Montague: I guess when it comes to music and judgement I am very old-fashioned: “Judging” is NOT a popularity contest. In fact judges are NOT performers, they are not hired to be put in place to receive performance feedback. It is the reverse with their focus to first rank contestants in a fair way, one to the best of her ability and second, provide a brief summary to support decision-making. As a competitor, the last thing I want is a judge sitting in judgement who continously wants, “….on-going development opportunities….[and] CONSTANTLY be evaluating their own abilities and competencies”. An insecure, hand-wringing nincompoop are qualities least admired in a judge and yet qualities reinforced by behaviours you OUTLINE. AB: my metaphor may’ve been stronger but I still don’t like the idea and see problems galore. I mean, if two piping judges, for instance, standing at different ends of a circle can come up with different pipe section results (and quite rightly and fairly based on their stance) then how can any audio recording after-the-fact or judging assessor in the crowd on-the-day come close to judging a judge’s judgement. That is one micro example.
With 87% feeling that there should be a feedback system, it’s probably a good idea to develop one. Right now competitors can only lodge a formal complaint to provide feedback on a judge, which of course is completely negative. I would think that a anonymous survey system asking competitors to rate adjuidicators 0-to-5 on things like “Overall, how would you rate the quality of this judge’s score sheets?” or, “Overall, how would you rate this judge’s ability to make the right decision?” would be useful. If they feel they don’t know, then respondents could have some kind of N/A option. Results could be averaged, and if there are significant trends – positive or negative – they could be considered, and the adjudicator then being notified and allowed to improve in specific areas of concern.
That said, doing all that is a massive investment of time, and a lot to ask of volunteers. The ROI has to be there.
I am not sure anonymous internet polling data necessarily equates to good ideas. Mind you, there is a school of thought that believes good law-making should be made by referendum (I don’t agree). I’ll sign off on this one with a last whole-hearted “bad idea”. Always the outlier it seems.
I think we sometimes expect too much from the whole thing. The vast majority of judges, even the ones who make bad decisions, are trying to do their best work, and doing it for not that much money. If the pipe band associations are doing their job, then the judging should be pretty good, on balance. Clearly, there will always be judges who, despite their own performance ability, are weak judges. And, there are those with less performance ability who turn out to be pretty good judges. The criteria and exams should help weed people out, and give others the insight and vocabulary to do the job. The weaknesses come in some associations where people are judging things such as ensemble with no recent experience, no real training, and no vocabulary for even writing an ensemble sheet. The associations/games should be looking more closely at who they are assigning to what, and judges should be more upfront about what they are really capable of doing.
Feedback from competitiors in pipe bands would be different than the CPA example. In the CPA example, all the competitors have reached at least a certain level of competence and understanding in terms of their art. They are “professional” players. In pipe bands, the vast majority are lower-grade bands, not always with capable leadership or understanding of the music. I say continue to develop what we have. The bands can offer feedback on judges at anytime by writing a note to the music board, and associations can continue to monitor results, offer education, and train new people. I think that for the most part, it works.
MG in my opinion is right on the mark. There is no positive to judging judges. The outcome would be 100% negative and also in poor taste. The 87% for this idea could have come from contestants who haven’t had the results go their way or because of association politics or personal vendetta, or by people who don’t compete or even play. The judges are invited based on their proven abilities, track record etc. Their task is hard enough at times without having the added stress of being provoked about his/her performance. Judges are rarely given the kudos’ except by the contestants who win or place. The remainder could be the sore loser’s or the lynch mob who then have nothing better to do than cause havoc. If this idea went ahead, it may be possible that judges would refrain from judging. Not because they don’t have the confidence in their abilities but in the end would it be worth having your integrity and good name in question? A guilty man who has later been found to be innocent will always be in question and found by many to be guilty, regardless of what the truth is.
In order to have the art evolve, both the players and the judges need to adapt and change. How that is done is a matter for debate, but it must be done. Otherwise stagnation will result. In relation to the judging panel, it needs to receive feedback from somewhere, whether it is the players or some other format.
I think that the term “feedback” as used in the question, can mean more than just a rating of judges, but can also include a way to contact and have an open communication with judges to further discuss a score sheet or incident. After judging Maxville in 2008, I recieved a question from a lower garde PM (that came to me via the PPBSO) to explain a comment I made on the bands piping sheet. I welcomed this opportunity to expand on my comment, thanked the PM for the interest, and suggested to the PPBSO that they be a ground breaker and open up lines of communication between competitors and judges. In this day and age of emails, blogs , ect, this would not be hard to due. I am also of the belief that we should be evaluated on a regular basis, and I would like the PPBSO to be the leader in this endevor and like the peliminary guidelines suggested by Andrew as a start.
I agree with most of Michael Grey’s comments. This is a very tough nut to crack for a whole lot of reasons, both good and bad.
When I saw the survey, it was my intent to notify Andrew that the EUSPBA has had a judge complaint policy for many years (check the EUSPBA’s policy manual on their website, page 18). This policy has been used on a number of occasions but most of the time, the end result was “no decision.” These “no decisions” satisfied no one which again reenforces MG’s comments.
One would “assume” even the most honorable judges would recognize they are NOT infallible. Unfortunately, based on my experiences, that is not always the case. Besides, what is a valid error or mistake in judging anyway? Let’s not forget the potential lawsuits either. Oh well, at least some are expressing their opinions but I really don’t see this going anywhere.
Who’d be a judge these days with these ever-increasing calls for justice, fairness, consistency etc being put out there? Do we want people to fill the judging ranks, or don’t we?
It seems to me like they have to jump through more hoops than we competitors already. Just ask all the o/s judges who weren’t there in August!
This is a highly/totally subjective art-form. The reality is that in a field of 15 bands, only one of them will 100% ultimately feel the judge got it right on the day. It is human nature.
Here’s a radical idea. Turn up, play as best you can, have fun, accept the result (good or bad) with style and grace, and we’ll see you at the next contest!
Thinking about things is a sign of security (insecurity would be characterised by angsting, or, obsessing). As is wanting to do things better. As is having the guts to receive feedback, positive and negative, and think about it, with the aim of improving things. Maybe the two words MUSIC and JUDGING don’t sit well together. How do you ‘judge’ art? Can it be judged?
this debate is pointless – Associations, or individual games with the blessing of the associations, hire judges. Associations are run through those elected by the members to act on their behalf (basic democracy). In effect the associations are determining which judge they hire through a process, either formal or informal, regardless that judge is already being evaluated. As such you could conclude that through the democratic process judges are already being evaluated.
Really though, what does it all count for, it’s a hobby with minor financial impact and major ego impact. Judges are underpaid for their knowledge, overworked and generally underrespected. To all those who harp on judges, check your ego and be thankful someone is even willing to offer some comments on your squeakings.
As an adult I paid $10 for a solo event last year for comments, cost to participate in the kiwanis music festival as adult is $30 per event…..you do the math and tell me that our judges are not underpaid.
I have to come on out on the side of Michael on this one. Judging is subjective, albeit within supposed parameters, and assessment of judges would most likely be the same. Competitors would probably be positive towards judges who favour the competitors’ own bands, and negative to those who tend to put them down. I think it’s known as human nature. We do operate in a democracy, and if we are really interested in change and improvement, we should lobby our representatives to present our opinions at local, national and international levels. The procedures are already in place, but action would require effort, whereas moaning doesn’t…..so we moan.
Norrie Thomson
I think where it gets really difficult is when a judge is also one of the power brokers in the association, and is also a fee collecting tutor of bands who he is judging. This tends to cancel out most of the checks and balances that are already in place. This is especially possible in smaller associations, where the talent pool is a bit thin.