Jay: Hello, good evening, good morning, good afternoon, depending where you are or when you watch this recording that we'll put out for everybody later. But welcome to the Accessibility Internet Rally of 2022. We are at the rally checkpoint.
So that means we are officially at week four, which is just about halfway for us this year, I think we're actually clocking like nine weeks, so you get a little booster week there. But welcome everyone. This is really for you as the team of developers to ask our Lead Judge, Rob Carr, who's able to join us today, some questions, anything you want to know about the judging form, the process, things like that.
But before we get to that, I do just have a couple of quick announcements and reminders for everyone. So just to start off, as I stated before, this is being recorded. We recorded this morning's session, so that way I do a little editing magic and I combined all the questions from this morning, as well as this evening. And I will put them all into one video so that way everybody gets to hear what everybody's questions were. So we'll have that.
We also have a special guest with us today, Sarah. So for those of you who have read your weekly emails, you may have noticed that AIR is part of a case study this year. We're very excited to be working with the University of Southampton, and Sarah is one of the researchers on that project. And so Sarah, if you just want to say hi and do a little quick intro for everyone.
Sarah: Hello everybody. It's midnight with me, so it's literally just turned tomorrow. I'm very glad to be here. Yeah, we're doing some research looking at the pedagogy of learning accessibility, of teaching accessibility, so we are really delighted to be involved and looking at some of the processes going on. If any of you have questions about what we're doing, please do reach out to me. You'll see the email was in the last newsletter. I won't interrupt the flow of things, but it's great to be here. So thank you everybody.
Jay: All right.
Julieanne: Sarah, do you want to drop your email in the chat as well?
Sarah L.: Will do. Thank you.
Jay: All right. So I will just cruise on along to our next slide here. So just a few reminders for you all. We will have our usability testing available to you through our Access Works program. Please make sure that you get those testing forms filled out by Monday, October eight... try that again, Monday, October 10th. There we go. If you have already filled out your form, thank you. If you haven't heard from us yet, don't worry, we'll start making those emails and connections with you next week. It's just easier usually for us to get everybody's info, so that way we can get a table going on the scheduling, so.
Erica: I have a...
Jay: Yeah, yeah, go for it.
Erica: Okay. So as Jay said, thank you so much to everyone who signed up. This is one of the folks in our Access Works program. This is one of their favorite things to do because they don't get to work with people who are learning pretty frequently, and then they get a chance to talk more in depth about who they are and their assistive technology. Plus, the nonprofit sites are pretty interesting.
For those of you who have not filled out the form yet, if you're interested, please do so. If you are on this call and your team has already filled out the form, we had a couple folks who had a fairly limited timeframe where they could take a test participant. If that is your team, not to worry, but we're probably going to be following up with you just to see if there are possible backup days, just in case we are not able to find a participant for your first choice. So if you can share a couple options, that's more helpful. But we're very excited to have some folks signed up and it's going to be a good experience, we're hoping.
Jay: All right, if you have some additional questions on usability testing, I'm going to ask to hold off on those until the end, just because I want to make sure that we also have time for our Lead Judge, Rob, today. And I know that he might need to bounce out a little early this evening. So I'm going to run through the rest of these reminders pretty quickly here.
Just a reminder about our final countdown that is coming up November 13th. The Zoom event will be at 5:00 PM Eastern Time. You'll register through Humanitix, as soon as we have those tickets finished up, we hit a little glitch in Humanitix, so the link wasn't ready yet today, but we will get those links out to you through our weekly emails.
You are not required to attend the final countdown, but it really is a fun time. We have the teams show videos of their journey. The judges and the rest of the volunteers will usually do a little countdown clock for us, so it's kind of a big celebration for everybody. But at the end, at 6:00 PM, is when your submissions are due. So we will provide copious amounts of instructions in those weekly emails. So don't worry. And if you have questions on those, we'll be able to take them later on this evening as well.
So with that, I am going to step back and let our wonderful Lead Judge, Rob , take it away.
Rob: Hey everybody. Great to virtually meet with everybody and it's perfect timing. Dog is now barking. We've had a hot water heater going in. Jay knows this, Julieanne knows this, all day. So now the dog's barking as they're finishing up work. What better microcosm of the era that we're in?
So yeah, my name is Rob Carr. I'm very pleased to be lead judge again for AIR this year. I am strategic accessibility coordinator at WebAIM and have been affiliated with AIR in some way, as a mentor or a judge, or this more coordinating role for, I don't know, seven or eight, six years. Time's a flat circle now. So it's hard to say exactly how long it's been.
But yeah, really pleased to be here today to just see what questions you all have. I mean, we can go high level, we can drill down a little bit. You might find me deferring somewhat to your mentors because there are some questions that they can help to parse through and really help to answer. But yeah, great to be here and glad I could make it, at least for a little bit. I do have an obligation, probably 25 minutes or so, but I might be able to stretch that a little bit too, since we're all here together, we can take advantage.
Jay: All right, so I do want to hit and I'll just kind of get the ball rolling, but feel free to put something in the chat, jump on the mic. We're pretty informal here and this is really just a chance for everybody to ask those questions that maybe you've been talking about and you're feeling a little lost. So I'm going to quickly scan through my Slack to make sure we didn't miss any from earlier today.
Rob: Oh, it was about the six pages?
Jay: Yeah. Yeah, that's what I saw.
Rob: So yeah, someone that asks, or had said the website must be six pages long, but does that also include accessibility statement, privacy policy, et cetera. And there's not any kind of requirement for things like accessibility statement, privacy policy to be part of that submission. So your six pages with minimal restriction are your six pages that you submit. So you should have a homepage and a contact page, but otherwise it should just be the pages that you want to put in front of the judges. So especially if you've built out forms, you've built in some of the things that are more optional that can get you extra points, then you'd want to be sure that those are in the fold and you can have as many as 10 pages as well.
Jay: And I did go ahead and put the rally rules in the chat as well.
Rob: Cool.
Jay: And that's what we're referencing, is the rally rules specifically state the homepage and contact page. Well Rob, while people are typing or thinking, are there any kind of tips or suggestions, maybe frequently occurring errors that you hope people can avoid this year?
Rob: The mentors need to be looking for that stuff, right? No, the things to look out for are those things that are on every single page. The way that we judge is we look at each instance of an issue. So like I mentioned with a broken link in a header, if you have a broken link in a header and that's on every single page, you're going to go down to no score for that one really, really quickly. So look for things in those repeated sections of pages. Make sure that your header and your footer, especially if they're the same on every page, make sure they're in good shape, because that is one way that teams lose points, and I think sometimes people don't think about the fact that when we do the judging, we're basically evaluating and looking at it like somebody with a disability. And so that's why we look at each instance of an issue.
It's not like a WebAIM report, or another evaluation report where you might say, "Hey, the footer has these issues", we will score it, or take points away every single time. So that's something to keep in mind, is to be sure that those repeating elements are in good shape. Other than the technical conformance and the evaluation criteria, I think that's probably the biggest tip, is to make sure you don't have these repeated instances of problems, just because you have the same thing in multiple even patterns. If you have a calendar widget that's on a couple of pages and it's really broken, that's going to take away a lot of points really quickly.
Jay: Right.
Rob: Everyone's quiet. Oh nope, go ahead.
Speaker 1: Hi. I wanted to know if you had the website from last year, the one that won. I don't know if you guys put that up, but it'd be great to just see what that site looks like.
Jay: Yeah, I think we sent... I want to say we sent it in one of our emails out to the teams. Let me pull those links up. Yeah, that's the winning site now. The one what's getting pulled up is their current site, so I could not tell you if that's exactly what was judged because the pages that they submitted to us were essentially on a different server that's no longer that we have access to. So what you're looking at is their current live site. And let me see if I can pull up the other links for the other two sites that got second and third, because I think those were still ones that are on our server and not necessarily their current live site, where they can continue to edit.
Rob: And that's another good point or at least another good point, Jay, which is if the site that we're going to be judging is live, it's in production, it's published out there, well ideally that's not the case, but if that is the case, try to make sure no one's making changes, because someone can easily go in and break content that affects the judge scoring because we're not specific about when during that, basically, month period people are going to be judging, it's when they're going to be able to work it into their schedule. And I don't know on y'all side, Jay and Julieanne and company, are we expecting to see sites in production environments?
Jay: Yeah, so we'll give instructions similar to last year, where we had sites that the organization is live, they have events going on, especially over Christmas. People are doing their end of year fundraisers. They're like, we can't have this site not be active. So we will have instructions in terms of how to give us those pages that you are submitting. So they are not live, they're on a production side of things. So that way the org can still do their business and edit what they need to publicly. But then that way the judges are getting things that are essentially untouched.
Julieanne: This is Julieanne from Knowbility, speaking as a judge in past years, I was actually on a site when it was being changed and it broke a few things. So you're trying to do it, and so if you're judging and you're going through and they're changing things, someone's changing them while you're going through them, it nullifies things that you've already done and you have to go back and start all over again. And that can create all sorts of problems for you both scoring wise and with a possibility of disqualification. So you want to make sure that we do have those static aspects on our server for the judging, so that it's not being changed as we're going through judging it.
Jay: Oh, and somebody pulled those links for me, thank you. From last year's sites.
Speaker 1: Great, thank you so much for that. This may be a hard one to... I don't know if this will make sense, but what stood out from the past years, those sites, that one, is there anything that they kind of had in common? I know there's that form, but I'm wondering what makes a winning site, a site that stands out from the rest?
Rob: It really does. This is the least creative answer, but I think it's the most honest one. It really does come back to making sure that you're paying a lot of attention to the judging form. And you mentioned that, and that's what I would really rely on to answer that question, because there's not a particular platform that people sit on that puts anything ahead or tends to knock people back. It's generally just the ability of the design and dev teams to hit the most marks. And so, some of the stuff like I talked about with making sure that if you have an issue, at least it's an isolated one, I think helps strategically. But otherwise it really is paying attention to the evaluation form.
And part of that is keeping in mind that the evaluation, the judging form, for this competition is not just like a big Web Content Accessibility Guidelines check. There are success criteria from the Web Content Accessibility Guidelines from Accessibility Standards. There are some that are not part of that, as well. And so I think that that's something to, just again, reinforce the idea that you should really get to know the judging form very well. And as you have questions, run them by your mentors, the mentors can always get a hold of me to discern anything that's in the judging form. But really that's the best answer, is to get to know that document really well, understand it and try to make sure that you're doing everything you can to design and implement components and content in a way that aligns with the different things in the judging form.
Speaker 1: Great. Well thank you.
Rob: Sure, thank you, you're welcome.
Jay: Yeah, so as you were talking about that, Rob, I was kind of flipping through it. When you look at that judging form, and we did put a link in the chat, there are, when you look at the form itself, you'll see one column, we'll say item. So it's just the item number of the judging form. Then you'll see a WCAG column and we'll put the where it's citing from those guidelines, and then the title of the item and then the description. But there are at least half a dozen or so, that are not tied directly to the WCAG guidelines that are more just kind of quality of life, I would like to call it. And making it an enjoyable experience for somebody to be on that site. And things like that. So yeah, I agree. Definitely that judging form's going to be your best friend. Oh, here we go.
Rob: Yeah, Patrick, that's a good question. Patrick, guys, do you have any tips on how to study the judging form? Well it's a little overwhelming when starting. So the way that we did make some changes, or actually my coworker made some changes to the form. And I would suggest just looking at it by tab in the form at this point, because you've got these different sections that we judge and we evaluate by. And that might be a way to break things up into a little bit smaller pieces because there is a lot there and it is, I think it's true of accessibility, when you're first really beginning to dive it can be pretty stinking overwhelming. So I doubt that you are alone. You're certainly not alone in the history of the competition. You're probably not alone in wondering that now, but that would be my suggestion, would be to take advantage of the fact that we have split the judging form sections up a little bit.
And so maybe try to get to know those one at a time, and focus on the concepts that are maybe the newest to you, to try to hammer through those and get a better understanding of what they're looking for. I think that once you get more comfortable with it, you'll see that a lot of stuff is related. There's a lot of stuff about keyboard accessibility that is, it's all split up into separate lines, but it still comes down to good fundamental keyboard accessibility. So there are ways that you can group it mentally. Jay showing the accessibility tab right now, I think that's probably the most overwhelming piece because there are so many checkpoints in there. So that one you might want to parse out a little bit more, and just look at things maybe in groups, even.
But so much of it too depends on your learning style, how your brain is wired to be able to compartmentalize things a little bit. But that's at least maybe a starting place. And I would say lean on your mentors too, really. The crew of mentors is always amazing, year in year out. So I would recommend leaning on your mentors to help to either come up with some ways to maybe go through it and parse things out into smaller pieces, or just to get a better understanding maybe of how they would group some of the stuff together. That's really, I think something that the mentors will be really invaluable in doing.
Jay: Yeah, Cindy's great. You've got a great mentor there. Yeah, please, I know sometimes teams will get into their flow and their rhythm and then we'll have mentors go, "Well I hope my team's going okay, haven't talked to me in a while". Please utilize your mentors. They are wealth of knowledge and experience and they want nothing more than for you to succeed. So always when you have questions, reach out to them.
Julieanne: This is Julieanne again from Knowbility. Also, if you're running short on time, your mentors can help you strategically plan what to make sure is tightened down really well for maximum points, if that's part of what you're looking at. So that's always a good place to lean into them is like where are we going to get the biggest bang for our buck if we're limited on time and the things we're trying to build out towards the end.
Rob: Yeah, I mean the judging form is really the guiding light, we've tried to make that as straightforward of a document as we can. It is broken up this year. For those of you who might have participated in previous years, we do have different tabs to try to make it a little bit easier to consume. But really, the judging form is, like I said, the guiding light for the whole thing.
There is a question in the chat, what should we be preparing to submit alongside the site? Sunday, would you mind expanding on that a bit? Because really as far as submission goes, what the judges will look at are the pages that you declare in that submission tab, where you've got to give us at least six, you can give up to 10, but that's really what we will be looking at. We're only going to evaluate the pages that are in the submission form.
Sunday: Okay, thanks. Yeah, I was just going to ask if there was anything aside from just the site pages itself, any videos or explanations of the design approach, or anything there that would be relevant from a judging or submission standpoint?
Rob: On the submission form, you do have an opportunity to point out some things, and I would take advantage of that. I think the Jay's showing that form now and where you can add some notes that could be helpful for the judges, especially with that extra effort piece. If there's a place where you all developed a custom component, or you had to go above and beyond to make something more accessible, that's a really good place to point it out and brag a little bit about some of the work that you all did. But really otherwise, no, it's just a matter of submitting the URLs for the pages, so that the judges again will know which pages to actually evaluate. It's not a site-wide evaluation, the judges will only look at the pages that you put on the submission form.
So and parcel with that is make sure you're getting the couple of required there, just a couple that you need to include, but make sure you're putting your best foot forward. Where are the best opportunities to gain points, and then maybe be confident that you're going to gain points with those and not end up with elements that are tricky and that might take points away. I see Sarah has a hand up as well.
Sarah: Yeah, I'm Sarah, I'm on team Getty. My other teammates couldn't be here, I'm so sorry. We're in Seattle so they're stuck in a meeting still. But I did want to ask, are there specific assisted technologies that you're using to test, or is it looking at more general accessibility since limited timeframe, we can't really do everything for all AT.
Rob: It's more the former, the judges come in with a lot of experience, for the most part, in evaluating websites for like WCAG conformance. And one of the caveats I always give is, we're not just looking at Web Content Accessibility Guidelines success criteria, we're looking at a handful of other things. And some of the checkpoints, if you read through the form and you know WCAG, see where we've kind of lumped some things into one checkpoint. But no, there's not a prescribed set of either assistive technologies to use or testing tools to use. We really do rely on the judges to bring their toolkits in. And one thing that could be insightful to help to answer that question is, we actually pair up judges who might be very new, we'll have a few new judges this year, with judges who have been judges in the competition in years past so that they can do some knowledge sharing and maybe help with those toolkits a little bit.
So no, there's not a specific set of AT that anybody's going to be coming in and testing with. And we actually have a decent amount of breadth, as far as experience and the toolkits that people have. I'm at WebAIM, we have judges from DQ, we have judges from Level Access, we have judges from private and public sector, and so people are bringing a lot of their own experience. And again, that's why, or that's largely why, we pair people up so that they can go through, use their tools and especially when they see a discrepancy, we ask them to work together and actually meet, ideally in a synchronous kind of way, to go over their results and resolve any of those things where they might be a little bit askew from one another.
Sunday: Hi. Sorry, it's Sunday again. I could not get the hand raised, so sorry to just interrupt.
Rob: No, you're fine.
Sunday: If I can jump right in with another question. We are working with a nonprofit that is using WordPress, and we're running into a couple of platform accessibility issues. We're looking into workarounds, but they seem to be built into the template. Will that be counted against us and are there any recommendations for how we can work around that? Should we be looking at different templates or something there?
Rob: So the judging form is the same for everyone, is probably the best answer to that question. And we've talked about this as a crew, gosh, over the last couple years, especially. How do we balance the NPOs preference maybe for a component library, or in this case a particular theme from WordPress, with the realities of the competition. And where we've landed, as far as the judging is, again, that the judging form is the judging form. So I think that in terms of thinking about alternatives, if it is specific components, if there's an inaccessible accordion, can you find an accessible one, or a more accessible one, to pull into the site? And remember, too, that this is a very focused thing and Ability has done quite a bit to try to make sure that the pages that are going to be evaluated and judged, they might not match production, necessarily.
So I think there's some leverage with that, as well, to be able to say, look, the page that has our exemplar, I'm going to stick with accordion because that's what I picked to begin with. The page that has our exemplar accordion is going to be in our submission form and that's going to be evaluated, as far as what the live site looks like. There could be differences there. And that's just I think a reality of the environment we're in, where WordPress actually does have a lot of accessible themes, accessible components. You have other platforms where teams are going to be a little bit more constrained, as far as what they can use. But we have landed on pretty consistently this is an accessibility competition, and so some of the creativity might not come as much in design or implementation, as much as it is in figuring out how to vet some of the plugins for accessibility before they go into the pages that are judged.
Jay: Right. Good question. And just to talk to what Rob was speaking of, I know earlier in the competition some people had ask for, "Hey, can we see the winning sites?" and things like that, and the winners from last year, we showed you their live site, which is what's currently on their site now, but in terms of what they submitted, that was on a different server that we don't have access to anymore because those were not necessarily live pages, it was just what was submitted. So it's one of those things where it's like, this is what was submitted, but after it's out of your hands and into the MPOs, you don't get to control that, necessarily, after they get their hands on their pages. Looks like you're not alone Sunday.
Rob: Right, yeah, as I was answering that, I thought it might be an answer for a number of people who were wondering the same thing.
Sarah: Yeah, it's hard. We were just debating with our team how much we want to be open to switching whole themes and stuff, because that's a lot to take on. And our team, specifically, had a hard time meeting up with the client for the first couple weeks, so we're just now ramping up and we're really going to push it on the back half of this competition. So yeah, Sunday you're not alone.
Rob: Well, and it's the way the web works now. When AIR kicked off, everything was custom built. I mean, it was all basically being done in a command line. And the modern web is not like that. You do have environments like WordPress and you have evaluation of plugins, you have these very different environments, but I think it's reflective of that. And so I think that, from a judging perspective, it's kind of cut and dry. But for you all, it represents some different challenges, because it's not so much that you're going to build the accordion from scratch, that you've got 150 accordions to pull from in WordPress. And so how do you evaluate to find the most accessible one of those, that you can implement in at least the pages again that you're judging, or that we're judging.
Julieanne: Hi, this is Julianne King from Knowbility. I just wanted to throw out to y'all that are submitting sites, if you've got working with NPOs that have live existing sites, what you're going to submit is something that is static. I'm sure it's in the guidelines somewhere, I haven't read that recently. I just wanted to let you know from a judge's perspective the importance of having static pages and not on a live site. As I've been going through judging previously and the site was literally being changed while I judged, and some elements got broken and I had to go back and redo the entire site, so it adds extra effort on the judges if you're breaking, it can lead to disqualification. And one of the things was is when they were making those changes, something broke and that actually reduced their score, unfortunately. But we just want to let y'all know when you're submitting, make sure if you've working from a live site, that you have one that can go that is static, that will not be changed between the time of submission to the final countdown, or not the final countdown, the award ceremony.
Rob: Yeah, I mean the judges are like everyone else, they're all volunteers, and so they're working this into their schedules as they can. So anytime from that submission deadline at 6:00 PM until the judging results are due in to me, they could be evaluating sites and it could be the beginning, end, middle. We don't control that. They have about a month to do the evaluations. So that is a really good point, Julieanne, to make, that as much as these can be locked down, and I do think that Knowbility is done a fair amount to try to make sure that they capture basically a snapshot like a Wayback Machine of the pages that are submitted.
But yeah, just do be careful if you happen to be an environment where the NPO needs the site in production, that's the only place it lives. Maybe some backroom deals to try to make it so that they're not making content changes, because as many of you know, one content update can break a heading structure for one of the pages that's in the judging pool, and that can end up really decimating the score. So I think that's an important point, Julieanne, thanks.
Jay: And Adrian, you got your hand up, and just let y'all know, we will have info on ways to do that.
Rob: Yeah.
Sarah: Thank you for that.
Jay: Not a problem.
Sarah: I did have a follow up, or not really a follow up, a different one for Rob , and it's just general curiosity. Do you judge sites in a certain way depending on the subject matter, thinking about accessibility of the audience. For example, my group is helping an NGO that is a community dance center, versus a community library, or versus an online gaming form. I don't know the different groups are, but do you judge based on who you think the audience is, or is it more of a general accessibility assessment?
Rob: We are agnostic.
Sarah: Okay.
Rob: We are taking, again, this really diverse pool of judges and putting a bunch of sites in front of them with a lot of Web Content Accessibility Guidelines success criteria, and some others that are not. And it's a very dispassionate process, for the better or worse.
But no, we're not generally thinking about the audience of the sites. I think if there are specific things about the audience you want to point out, we actually just had a conversation internally at WebAIM earlier this week about some content that was very audience driven and it maybe shifted a bit of the evaluation report results that we gave. So if there are things like that, point them out. But also take a very dispassionate look at the judging form, and keep that in mind as you're building the site out because the judges who are going to look at the site are going to be coming in from whatever background, and they're really going to be driven by the judging form.
Sarah: Gotcha. Okay. Perfect. Thank you.
Jay: And just a quick note, there is an item on the form, number 35, and that's called site appropriateness. We actually will contact the clients, they'll have a very short, little survey that they complete after you have submitted, for them to give us feedback and those scores get sent to the judges, and that's part of their consideration process in that.
Sarah: Gotcha.
Jay: So your client will not be dispassionate, though.
Rob: True.
Sarah: Okay.
Rob: True.
Sarah: Perfect. Yeah, no, that's okay. Perfect. Thank you.
Rob: I'll give a couple of tips, just based on having been around this for a little bit, is remember that each thing on the judging form is viewed on a per instance basis. So for example, if you have a color contrast issue in your footer, you're going to lose all the points, because that's going to be on every single page that you submit. So when you're going to be thinking about making up some points, make sure that whether it's a component, or it's region on the site, make sure those things are buttoned down. That is a way that the judges pick up some speed, and that's why a lot of teams lose a lot of points. So think about that as you're building things out, those things that occur in multiple pages that you submit, button them down as much as you possibly can.
Otherwise, yeah, the judging form is really the bible for the thing. And we've redone the judging form, it'sactually my coworker, John Northup, who has only been my coworker since March, but he put a lot of work into breaking it into multiple tabs. I think we actually owe, Jay, I realized it throughout the day, I think I owe you an updated, updated version that got some comments and such, so I'll look at that tomorrow, hopefully, and if I do need to send you an updated one, I will do that.
But the judging form itself, that's where everything comes back to. And the mentors, for the most part, I think are comfortable with the form. So your mentors are going to be your first point of asking questions. If you have technical questions about different elements within the judging form, start with them and if they can answer it, then it'll come to me, and I can put it out in front of some of the judges if there's a need to evaluate a response.
But yeah, really do you use that as your Bible. It is a lot. We had a question this morning that I know you're going to do a bunch of editing beautifulness, Jay, but I will address it again, because I thought it was a good question. How do we make this a little easier to consume? And my first answer to that is, ask your mentors if they have any advice. One of the things you'll find with the judging form, if you're new to accessibility, is that there are a lot of things that are closely related, but they might be dispersed across the form. So for example, if you think about keyboard accessibility is kind of a uniform concept, well that's represented in a bunch of different checkpoints. So, in terms of consuming this information for the first time, number one, look at it across the tabs that are in the form now, but the accessibility tab in particular, it's really big a lot there.
My motivating point to you is that you, if this is new to you and it's a lot, you're like any other developer or designer who's new to accessibility, where it's just a lot of stuff. So parse through it'll get to where it's more reflexive, even over this relatively compressed timeframe. And really lean on your mentors for clarifying points, because again, if there's something that they can't clarify, they're comfortable with, they can always reach out through Jay, or directly to me, and we can get that information back out to you all, through your mentors.
Jay: Right.
Rob: Everyone is good to go, Jay. Everyone's set.
Jay: Right.
Rob: You'll receive no questions, no comments, no concerns from the teams between now and the middle of next month.
Jay: I wish. I wish.
Rob: It's [inaudible].
Sarah: Yeah, you might as well just move the submission date up already.
Jay: Yeah, let's not do that.
Rob: Oh, there you go. Yeah, that's good. Give us more time. That's perfect. I'm literally still recruiting judges, because with everyone else, we're finding out that the people have full schedules now, that weren't full a couple years ago, so don't worry Jay, it'll be fine. Don't sweat it.
Jay: Oh, it always is. It always is.
Rob: Exactly. I'm glad you think that, too.
Jay: It always works out.
Rob: It always works out.
Jay: I'm a performer at heart, so it's always when the curtain opens, magic happens, no matter how disastrous the weeks before rehearsal have gone, so. Oh, just a quick question here from Tanika. Can we submit our sites early? You can. I know that last year we had some teams, because of time zones and things, they did not want to stay up until whatever ungodly hour it would've been for them to officially send an email at 6:00 PM at night for us. So that's totally fine if you would like to. However, judges won't start looking at anything until after that final submission deadline, so I know some people would send in their stuff to us beforehand because they just didn't feel like waiting or scheduling their email for that time.
Rob: I mean, there's zero practical benefit from submitting early other than time zone changes. So if it's the difference between running a couple of checkers over your site, doing some digging around in dev tools, don't feel pressured to submit early, is what I would suggest, because there's a bunch of nuance.
Adrian, it's the best question. What are the most common failures? So I'm a shameless self promoter, but also before I joined WebAIM, my answer would've been, look at the WebAIM Million. That's going to be the best indicator of the most common web accessibility fails, at least that are automatically detectable. There's a lot of stuff around keyboard accessibility that, I mentioned that earlier, kind of was a little bit of foreshadowing, keyboard accessibility stuff is really, really common. And that's not that simple of a topic. There are a lot of pieces to that.
But yeah, look at the WebAIM Million, see the things that we continued find that are automatically detectable. Think about keyboard accessibility. That's a really, really common issue where folks have issues. The one bit of insight I'll give you is, when you think about keyboard accessibility, think about different states as well, that is one place where we see stuff break a bunch of the time.
Yeah, Julieanne, for sure. Color contrast. So my coworker, Jonathan, has literally a three hour workshop on color contrast. So those of you who might diminish it, it's a big thing. It's a big topic. And that is one of the things, if you look at the Million you're going to see, that's a big fail point.
Sandy asked, what AT browsers are we testing with? We're letting the judges come in with their toolkits, so we don't prescribe a particular environment for them to test in. We know our judges, whether they're new to judging in this competition or not, I just talked to one today who works for one of the big private consultancies, and she does evaluation work. That's a massive part of her role. So we ask them to bring in their combos to work with. We suggest some tools for some, like Bookmarklet, that are a little bit more focused in what they do. But for the most part, we know that the judges were pulling from, they're skilled, they know what they're doing.
And again, we do pair them up, which is a way to help if someone is new to evaluation work, if they've been in accessibility for a minute, but they're newer to evaluation, then we always pair folks with a more seasoned tester and evaluator, to make sure that we fill in some of those gaps. So there's not a specific set there to work with. So my suggestion would be, make sure you're developing, designing fundamentally sound and accessible stuff. It should be fine.
Sarah: I have a question that I was going to look to see if it would been covered in emails, which I'm sure it has, but I'm blanking right now, so I'm just going to ask. The usability testing, is it to test our progress thus far on the website for the NGO that we're working with? Or is it to test their live site to give us some direction as to things that we should prioritize as we go?
Jay: It's on. Yeah, go for it, Erica.
Erica: I was going to say, it's going to be on what you're working on.
Sarah: Okay.
Erica: So, whatever, and your site doesn't have to be 100% complete. It just needs to be ready so that people could go all the way through a task flow. So signing up for something, or selecting an activity, something like that.
Sarah: Gotcha. Okay, perfect. But it's different than the judging, right? Since it'll take this before the site of submission, it's kind of just like a check-in point where you can get feedback, where it'll be taken into consideration with the judging.
Erica: It's not going to be, and Jay, correct me if I'm wrong, but it's...
Jay: It's not part of judging. This is a separate, this is an opportunity for you as developers, if you've never done a usability testing experience, for you to have that opportunity. Judges do not get any of that information. They don't get any of the reports from it, this is just for you, as the team, to have somebody test out your site and see what it's like for somebody to use your site. But judges don't have any, we don't even tell them if you signed up for it. We don't tell them anything. So it's really just for you.
Sarah: Gotcha. Okay. Got it. Got it. That makes sense. Thank you so much.
Erica: Someone who put a question in about Wireframes, I'm not going to come out and say no Wireframes allowed in your usability testing, but I strongly, strongly encourage you to, during that usability testing session to bring in the site you have in progress, the WordPress site. And the reason for that is because frequently Wireframes are not terribly accessible, just by their nature, they're all visual and not interactive. So not really tab navigation or screen reader friendly. So, not to say that, I mean, there are people doing all kinds of exciting research about can you make an accessible Wireframe. So it is not like a, don't touch this ever again. It's just think about this, when you got much more brain space to devote to that and focus on your website for the usability testing session.
Rob: And I'll just say, the judging form and the judging results are really, really helpful for designers and developers. But this usability testing is huge. So, if you're on the fence about it, get your team to get over the fence, because the feedback that you get goes beyond accessibility, goes beyond even what's on the judging form, that goes beyond the success criteria. Take full advantage. And yes, the piece about the Wireframes, any design artifacts, such a huge element of the way the web works and Knowbility. WebAIM, the private sector folks, we're all trying to get accessibility built into that stuff. So I think it would be great to see some of that stuff. And then from the judging perspective, maybe we can talk about, is that a part of the process earlier somehow? I don't know how to make it part of the process there, but I think there's a lot of value in looking at those artifacts, because it's going to save you all when you go to implementation, a ton of time.
Jay: Oh, there's a little ta-da.
Rob: There's a party.
Jay: There we go. You know what that tells me, it's time for Accessibility Jeopardy.
Rob: I'm going to have to hop off, I'm afraid. So thanks to all of y'all who are part of the teams, we really look forward to having your submissions and to score the sites. We wish everybody the best of luck. And like I said, lean on your mentors, if they don't have an answer, they can always reach out and get to me and then I can get to the pool of judges if there's something that's kind of dicey. So thanks for all the time you're putting in. It's amazing, this is one of my favorite parts of my year, every year. And best of luck everybody.
Jay: All right. So I'm going to go ahead and wrap up for the evening. Thank you everyone for joining us. We will see you again in our Zooms for our final countdown in November. You will be getting our weekly email, probably Tuesday coming up, where we'll have not only the link to this evening's recordings, but some additional reminders about the Rev captioning deadlines. I can't remember what else we were going to put in there off the top of my head, Julieanne?
Julieanne: Videos.
Jay: Oh yeah. So just a quick little thing about the videos. They are not required. This is just an opportunity, if you as your team want to do a little video, usually they're two or three minutes, maybe five minutes long, where you just tell your story. Sometimes they'll highlight certain features, they'll talk about their journey, maybe they'll review the website. So it doesn't have to be anything long and complex, like I said, it is not part of your submission, it is not required. But it's just a nice way for everybody to share their story and that way we all get little sneak peaks at your sites, and things like that, so we do encourage it, but not required. But we'll give you some more info on that in the weeks to come. But thank you everyone. Have a lovely, lovely rest of your day and we'll see you next time.