Judging Training (Circle, Routines.. etc)

Working on a footbag-related project? Ask for help, get feedback/advice here.
Post Reply
jay7
Flower Child
Posts: 1616
Joined: 22 Jun 2006 11:31
Location: Europe

Judging Training (Circle, Routines.. etc)

Post by jay7 » 08 Jan 2017 10:50

Greetings modified.

Spawned by the Matt Kemmer discussion regarding an automated circle judging system, we argued also in favour of trusting humans to simply be trained better in order to produce better tournament results. An idea I had at last worlds which I did not act upon, was the concept of a training program for those who would judge, which better enforces regulations and better ensures that people follow the rules.

Ideas for the program, in a brainstorm fashion:

1.) Given that footbag is so dispersed that judges will not be dedicated, nor attend all areas implies two things:
a.) Either we need online judges who "travel" to each tournament via video, who are held to a certain standard to judge,
or
b.) We ensure that regardless of region, the tournament director needs to ensure that the people on the ground are indeed trained.

I personally feel that "a.)" is the most reliable way to go, given my actual on the ground tournament experience... In a perfect world these judges would fly to the location a week in advance, get over jet lag, and be all sober and attentive for game day - however obviously in Footbag this is not the case, but limiting the traveling, jetlag, and atmosphere/party/staying up late/etc factors are mitigated by the virtual presence. On the other hand, you would need good quality assurance that the video is properly done, (i.e. ensuring it is actually taken, and has the players and bag visible, etc), properly sent, etc. This is a big risk, in case a phone gets lost, or what have you.

"b.)" is a possible alternative, but is subject to a large array of issues. Finding dedicated judges, getting that valuable tournament time to sit people in a room and train them, ensuring that these random locals have enough consistent tournament experience to be up to date, etc etc.

Either way, "a.)" or "b.)", a universal training program could be developed. An idea being:

1.) Have a video, and e-booklet. The video would show examples of variety, density, etc, and cover the rule book.
2.) The instructor (recorded or not) would then explain these points, answer questions, etc.
3.) The "judge to be" would then fill out their e-booklet and be somehow coached on their results.

As it stands today, somehow I feel that the rule book alone does not enforce this feedback loop, to ensure that the person took in the rules as intended and can actually act on them in a standard way worldwide, at every tournament. Having just a 15-30 minute video could boil it down, and having a nice grading system with feedback could ensure everyone is at least in the same metaphorical chapter, if not "same page" on how things should work.

-----------

So, the question is: (Assuming we don't go with a calculated system and bypass all the human error)

1.) Are there any disagreements that the proposed idea above would help our sport's judging unanimously?
2.) Assuming the above is 75%+ in line with a good idea and the idea is worth improving, are there any better suggestions, or improvements on the above?

Thanks
Jay Boychuk

User avatar
Tjuggles
Flower Child
Posts: 1645
Joined: 27 Mar 2005 20:03
Location: Wellington, NZ
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by Tjuggles » 08 Jan 2017 16:05

I feel like the physical absence of judges would take something away from the tournament. It would almost be like an online competition instead of a tournament.
Jay wrote:Having just a 15-30 minute video could boil it down, and having a nice grading system with feedback could ensure everyone is at least in the same metaphorical chapter, if not "same page" on how things should work.
One addition that I think would be beneficial is to have judge norming sessions, in addition to the instructional video. The sessions would occur before a tournament (and after the video) and would involve the to-be judges watching N videos of circle competition and scoring them individually. Afterward the judges have scored individually, they come together as a group discuss their answers. A moderator controls the flow of the session and addresses issues which arise from, for example, judges giving very different marks, to mitigate the effects of subjectivity by settling areas which are unclear in assigning a score. Doing this allows for each judge to

1. gain experience with the system,
2. gain clarity regarding how others assign value which can give them confidence that the scores they are giving are valid, and
3. help to ensure that the judges are scoring in similar ways.

A system such as this could create a systematic methodology for training circle comp judges.

One advantage of this method is that each year the to-be judges, by watching at least one video from a previous norming session, are able to compare their answers with the previous judges' answers. Among other things, this allows for the judges to be sure that they are giving similar marks that previous judges gave. This means that each year the circle comps are being judged by people who give scores in a similar manner to previous judges, increasing the reliability of the judging system as a whole.

I have experience doing this in a university setting with high-stakes testing and could give advice, or perhaps even run a session. Or do nothing. Multiple-choice bitch.
TJ Boutorwick

"You can do anything" -Greg Nelson

boyle
Post Master General
Posts: 2795
Joined: 19 Sep 2007 23:42
Location: Canberra, Australia
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by boyle » 09 Jan 2017 07:46

Yes, this kind of moderation could work well, but maybe there should be someone designated as more of an overseer, scrutineer...to make sure things have veen done right. A video referee of sorts.

User avatar
Tjuggles
Flower Child
Posts: 1645
Joined: 27 Mar 2005 20:03
Location: Wellington, NZ
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by Tjuggles » 09 Jan 2017 15:04

boyle wrote:
09 Jan 2017 07:46
Yes, this kind of moderation could work well, but maybe there should be someone designated as more of an overseer, scrutineer...to make sure things have veen done right. A video referee of sorts.
Do you mean to make sure that the scoring has been done correctly? Do you mean after the judges give their scores during a tournament, before the results are made public?
TJ Boutorwick

"You can do anything" -Greg Nelson

boyle
Post Master General
Posts: 2795
Joined: 19 Sep 2007 23:42
Location: Canberra, Australia
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by boyle » 10 Jan 2017 04:07

Yes

User avatar
Tjuggles
Flower Child
Posts: 1645
Joined: 27 Mar 2005 20:03
Location: Wellington, NZ
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by Tjuggles » 10 Jan 2017 15:41

I see
TJ Boutorwick

"You can do anything" -Greg Nelson

boyle
Post Master General
Posts: 2795
Joined: 19 Sep 2007 23:42
Location: Canberra, Australia
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by boyle » 06 Feb 2017 00:56

I've been planning on looking into this for ages, but haven't had the chance. In this thread - viewtopic.php?f=28&t=23901

there has been a notation of all the runs from circle contest finals in Copenhagen (Worlds 2015).

My simplistic analysis is this - who do any combo of 5>5 (and above) in the density round.

There were a lot less than I expected

Milan's first run was pretty long, but no 5>5 combos.
Jim's run was really short, but included BW>BW>PS Whirl (3 total)
Vasek's first run was very long (49, with 17 5 add moves) and included: scorpion's tail>motion sickness>motion; spinning whirl>whirlwind>spinning butterfly swirl; mobius>food (8)
Evan's first run had a couple of longer fearless combos. That's two sets of 5. (Ripped Warrior L > Blurry Whirl R > Blurry Whirl L > Blurry Drifter L > Blurry Whirl R; Paradox Symposium Whirl L > Spinning Whirl L > Spinning Whirl L > Spinning Whirl L > Spinning Whirl L) (10)

Round 2

Milan - BW>PSW (2, 2 total)
Jim - nothing, but a really long run (63 contacts, 12 5+ contacts) (3 total)
Vasek - also nothing (8 total)
Evan - Alpine BW>Gyro whirling swirl (2, 12 total)

Round 3

Milan - Ripped Warrior R > Paradox Symposium Whirl L > Mobius R (3, 5 total)
Jim - Blurry Whirl L > Blurry Whirl R; Paradox Symposium Whirl L > Spinning Whirl L (4, 7 total)
Vasek - Stepping Whirling Swirl L > Blurry Whirl R; Blurriest L > Blurriest R > Superfly L > (5, 13 total)
Evan - Blurry Whirl L > Blurry Whirl R (2, 14 total)

Interesting - Evan tops that count, but that's just a minor detail. The major detail was that he only had about half the contacts of some of the others, but it's interesting when thinking about planning how to approach competing in circle.

User avatar
C-Fan
Rekordy Polski
Posts: 11343
Joined: 23 Jan 2003 23:51
Location: Denver
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by C-Fan » 15 Mar 2017 10:24

BUMP a few months before Worlds. Any thoughts on identifying/training judges before Worlds?

User avatar
Wiktor
Multidex Master
Posts: 305
Joined: 01 Nov 2004 12:43
Location: Warsaw, Poland

Re: Judging Training (Circle, Routines.. etc)

Post by Wiktor » 19 Apr 2017 07:31

How to prepare judges? Get a serious panel before the event.
Make sure we have contact (and maybe BOP) counter from semis to finals.

Everone who wants to judge, can play online videos and pracitce alone, give scores live, without detailed analysis.
Than compare your results to real ones.

Again, most of the time, its not the wrong system, but human factor.

If you watch Copenhagen Routines, don't blame the system.
You can personally ask each judge why he/she scored this way and everone will have some strong points.
It's freestyle, so we never get it 100% risht.
In the era where players and not willing to compete at all, the systems shall be easy.

PS. IFPA Rules will be online shortly again.
Wiktor Debski

User avatar
C-Fan
Rekordy Polski
Posts: 11343
Joined: 23 Jan 2003 23:51
Location: Denver
Contact:

IFPA Judging Training Video?

Post by C-Fan » 24 Apr 2017 14:16

Good thoughts Wiktor.

The other idea I was kicking around, was developing a training video. Nothing fancy, just like 5 minutes explaining the theory behind Routine and Circle judging, then maybe a sample routine broken down. The idea being that judges could be identified ahead of time and they could watch the video training before a tournament, or if judges were IDed at a tournament you could make them watch a 10 min video to get a consistent overview of judging. The beauty of this is that it could be used over and over again at different IFPA tournaments, so we're not reinventing the wheel each time.

User avatar
Tjuggles
Flower Child
Posts: 1645
Joined: 27 Mar 2005 20:03
Location: Wellington, NZ
Contact:

Re: IFPA Judging Training Video?

Post by Tjuggles » 24 Apr 2017 15:18

C-Fan wrote:
24 Apr 2017 14:16
The other idea I was kicking around, was developing a training video. Nothing fancy, just like 5 minutes explaining the theory behind Routine and Circle judging, then maybe a sample routine broken down.
Didn't Steve G put together something like this? I remember seeing a video about him talking about how to judge, or something like that.

I think watching the video is a great idea. However, I think that it is important to put that knowledge into action through norming, i.e., actually scoring some routines and then comparing scores with other potential judges, talking about any differences in scoring, and also comparing those scores from potential judges with what the actual routine score was. It is 'easy' to watch a video and afterwards think 'yup, I got it, ready to judge', but perhaps not 'easy' to actually do it.
TJ Boutorwick

"You can do anything" -Greg Nelson

User avatar
Wiktor
Multidex Master
Posts: 305
Joined: 01 Nov 2004 12:43
Location: Warsaw, Poland

Re: Judging Training (Circle, Routines.. etc)

Post by Wiktor » 23 May 2017 13:17

Sorry, im in 10000 footbag projects at the same time. forgot about this topic.
So we have the videos both for singles and doubles:
http://www.footbag.org/gallery/show/13248
http://www.footbag.org/gallery/show/17137
The links may be dead for the moment, but once Steve makes all servers up, shall be here.

Anyway, the judging clinic should be a goal this year. lately we had no manpower ro run it.
I guess this year we can. Let's try to schedule it, and later lets find spokesman/leader who will host it.
I may help, but not lead. Steve the same. I'd aim for guis like Aleksi or Gordon or Peter Irish etc.
Can you guys ask one of them or other to participate.
Cool thing is to watch online routines together, comments, give scores and THAN discuss.
A clinic workshop. Not just a video.

Thanks,

Wiktor
Wiktor Debski

User avatar
C-Fan
Rekordy Polski
Posts: 11343
Joined: 23 Jan 2003 23:51
Location: Denver
Contact:

Re: Judging Training (Circle, Routines.. etc)

Post by C-Fan » 30 May 2017 06:19

I love these idea Wiktor. Can't wait to watch the videos when they're up.

Post Reply