Monday, May 25, 2015

Podcast Episode 10: Post-Murph Chat and Regional Update

Anders and old pal John Nail catch up post-Murph on Saturday afternoon.  Unfortunately the iPhone caught a case of Rhabdo about 7 minutes in, so the conversation gets cut a little short (but SWEET!).  To make up for the lost recording, Anders wraps things up on Monday with some thoughts on the first two weeks of Regional competition.

Check for week 3 of regional predictions Wednesday or Thursday this week.  Enjoy!

Thursday, May 21, 2015

Regional Predictions Week 2: California, Pacific and East Regionals

One week is in the books, and things went pretty well from a prediction standpoint and a viewing standpoint.  Before we get to predictions for week 2, here are some thoughts on what we've seen so far:
  • From a viewing standpoint, I thought the events improved throughout the weekend.  "Randy" wasn't bad, but to me, any workout with just a single movement just isn't quite as fun.  "Tommy V" was simply too many rope climbs to really be a great spectator event.  The long chipper on Saturday was pretty entertaining once the run ended (Emily Bridgers and Anna Tunnicliffe had a nice battle in the Atlantic).  The handstand walk followed by the snatch was OK, but nothing great.  But Sunday was good for some high drama.  The handstand push-ups on event 6 really caused some serious shifts in the leaderboard and it gave us some good battles, like Camille vs. Natalie Newhart in the South.  Event 7, while in my opinion not quite as good of an event as last year's pull-up/OHS workout, provided some great theatrics.  The men's final in the Atlantic was the only event that I had to watch live, and I (along with with plenty of others it seemed) was rooting for Elijah Muhammad to finally get his trip to the Games.  His comeback on that event was exactly the type of drama the sport needs.
  • In the past couple weeks, I complained a bit about the fact that only two attempts were given for the max snatch, but my reasoning was that it left athletes vulnerable to being penalized significantly for a single mistake.  But watching the event, I also felt like having only two attempts made for an awkward viewing experience.  I think we're trained to expect three attempts on something like this, and it felt odd that the event was over after only two tries.  It just moved too quickly, in my opinion.  If there were three attempts, I think you'd get more athletes really pushing the envelope on that third try, but generally we got some pretty safe lifts for most athletes.
  • Speaking of the snatch, for men, the average load lifted was 236 lbs. in the Atlantic and 226 lbs. in the South, and that includes 2 lifters in each region who had no good lifts (those counted as 0 lbs.).  For women, the average were 151 lbs. in the Atlantic and 144 lbs. in the South, including 1 athlete in the Atlantic and 3 athletes in the South who had no good lifts.  By comparison, the averages on the hang snatch event last year were 224 lbs. for men and 137 lbs. for women.  Remember, those fields included about twice as many athletes, so you'd expect the loads not to be quite as high, not to mention the fact that the lift was required to be from the hang and power snatches were not allowed.  I think it's safe to say athletes were not as close to their maxes this year.
  • The predictions were generally pretty decent in week 1.  The only major surprise qualifier was Whitney Gelin from the Atlantic, who I had pegged with a 1% chance.  Otherwise, every other qualifier had at least a 13% chance of qualifying.  Of the 9 athletes I predicted with greater than a 50% chance, 7 qualified.  After the regionals are over, I'll update the calibration plot that I showed last week.
  • For this week's predictions, the only significant manual adjustments I made were to boost Mat Fraser's chances a bit (not quite the same amount as I boosted Camille's last week, but close) and to boost Kara Webb's chances (I treated her as if she was a top 15 Games finisher).

One final note: I'm planning to record a podcast this weekend with John Nail.  We'll be doing the podcast after the Memorial Day "Murph" workout at my old gym, and we'll be chatting about regionals, my (still unfinished) road back to being a full-fledged CrossFit athlete again, whether Hamm's is truly the best cheap beer available and other subjects of vital importance.

And finally, predictions are below.  Enjoy week 2 of Regionals, everyone!

Thursday, May 14, 2015

Regional Predictions Week 1: South and Atlantic Regionals

In each of the past four years, fans who've watched the CrossFit Games have seen some really cool stuff. At the Games, you get to see athletes put up some insane weights (377-lb. overhead squats, for instance), you get crazy-brutal events like a 2-hour triathlon or the burden run, and you get to see some events with excellent finishes (like Josh Bridges holding off Rich Froning in "Push Pull" last year).  But what you don't typically get is a lot of drama.  With the exception of last season's men's competition, the men's and women's titles have basically been decided by the time the last event rolled around, and even in that men's final last year, you never felt like Mat Fraser could really pull it off.  You have to go back to the 2010 men's final to really get a big shift at the end, and even that was weird because it was hard to tell that Graham Holmberg was actually going to catch Rich as you were watching.

No, the Games is not where you go for drama.  That would be Regionals.  If you've been to the event in person on a Sunday, the tension in the air is unavoidable.  Watch Sevan Matossian's documentary "Only Three" about last year's Central East Regional and you'll see what I mean.  For that reason, the Regionals have long been my favorite part of the season as a spectator.

For the third year, I'll be posting predictions for each athlete's chances of making it to the Games.  In the past, these predictions have been fairly well-calibrated, meaning that, for instance, athletes with a 50% chance of qualifying in my model do typically make it to the Games around 50% of the time.  That doesn't mean the model is perfect, of course.  If it were, I'd give a 100% chance to the 5 athletes who were going to make it.  But then again, no model is perfect, and if there were a perfect model, well, there'd be no drama.  So keep in mind these are all in good fun.

For more background on the model and how it works, listen to the CFG Analysis Podcast Episode 8 or read up here.  In theory, the model should be getting better each year as we get more data from past years to help calibrate it.

Before we get to the predictions, here are some general thoughts on this year's Regionals:
  • As far as loading goes, this year is roughly the average of the past four years.  The required weights in the metcons are lighter than last year, but there is more lifting overall (48% vs 37%).  Although the exact number depends on the numbers we see in the snatch event, the LBEL looks to be around 0.72, which is lower than 2012 (0.92) but higher than 2011 (0.68), 2013 (0.60) and 2014 (0.58).
  • Olympic lifting movements play a huge role, even more so than in the past.  Every single lift falls into the "Olympic-Style Barbell Lifts" category that I've defined, and even if you restrict that definition to only snatches, cleans and jerks, that's 36% of the points.  Thrusters, overhead squats and sumo deadlift high pulls make up another 12%.  Historically, this entire group of movements is worth about 33% of the points at Regionals.
  • Burpees are nowhere to be found for a second straight competition after appearing in every competition from 2008-2014.  Very curious on the reasoning from HQ here...
  • I like the move to the Games-style scoring system.  This won't punish a single poor performance quite as much (Sam Briggs would have made it to the Games last year under this system), and it rewards elite performances on certain events more so than the past Regional system.  More old school thoughts on the scoring system here.  If I had my druthers, this is the scoring system I think I'd go with.
OK, well let's get to the predictions.  The only manual adjustments I made this week were to bump up Camille's chances a bit (despite her being in the highest-rated cohort already) and to put Sam Briggs into a slightly higher cohort (equivalent to athletes finishing below 15th at the Games last year but top 40 at Regionals last year).  Enjoy the weekend everyone!

Wednesday, May 13, 2015

Podcast Episode 9: Regional Preview (with CrossFit Naptown)

Today Anders is joined by Peter Brasovan and Jared Byczko of CrossFit Naptown in Indianapolis.  Both competed on Naptown's Games-qualifying team last season and are back on the Regional-qualifying team this year.  The guys discuss the 2015 Regionals, including thoughts on whether anyone will fall off the True Form treadmill, dealing with challenging judging standards, the importance of strategy in this year's events and whether anyone who misses the 6:00 time cap on "Randy" will make the Games (hint: NO).

Be on the lookout for Regional Predictions for the South and Atlantic Regionals tomorrow evening (Thursday, May 14)!

Wednesday, May 6, 2015

Podcast Episode 8: How Do I Make Regional Predictions?

Today's podcast covers the method behind the madness of predicting each Regional athlete's chances of making the Games.  I'll post the week 1 predictions early-mid next week, and we'll have another podcast talking about the events themselves.

Since I never posted it last year, here's the final calibration plot of how last year's predictions ended up.  Things shaped up pretty well.  The mean-square error was 4.02%, compared with 4.43% in 2013 and 6.07% if you gave every athlete the same chance of qualifying.

Wednesday, April 15, 2015

A Look Back at the 2015 Open: Part II

Welcome back to Part II of the 2015 Open Recap.  Sorry for the delay, but we have a lot to get to this year.  The introduction of the scaled division has added a whole extra layer of complexity, but I think there's some interesting things to be learned, so let's get to it.

First of all, thanks so much to Sam Swift for pulling the data for 2015 (as well as 2012-2014, which I have used to a lesser extent in this post).  He also has some cool analysis on his page, and I suggest you check it out at some point.

For each portion of this analysis, I had to decide whether to include scaled competitors or not. Often, I excluded anyone who scaled any workout, so as to be more comparable to what was done in the past (when scaling was not an option). For instance, with the correlation between workouts, it did not make sense to include scaled competitors at all, as this mixed in scaled and Rx'd workouts and made for an apples-and-oranges comparison. Other times, however, I did include athletes who may have scaled some workouts. For instance, on the 15.2 vs. 14.2 comparison, I included all athletes who did 15.2 Rx'd, regardless of whether they did other workouts scaled. I'll try to note which population was used in each section.

So now for the results.  I'll start with the correlation analysis that I've done many times in the past.  This basically tells us which workouts were highly correlated with success across the board in that season. The charts below show the correlations for women from 2012-2015 (the mens' results are quite similar).

We see first of all that 15.2 was a pretty solid workout.  Interestingly, it had a higher correlation this year than 14.2 did last year. You can see the same pattern between 12.4 and 13.3.  My guess is this is due to returning athletes having a better feel for this workout, and since returning athletes typically do better in general, we see higher correlations.

What is very intriguing is the fact that 15.1a, the max clean-and-jerk, had a relatively low correlation for the entire field, but a relatively high correlation when we limit to the top 1,000 overall finishers. This would indicate that for the top athletes, the best-of-the-best perform quite well in a max clean-and-jerk. But for the entire field, this event didn't pick out the top athletes as well as the other events.

The scatter plots below, which include an evenly distributed sample of the entire field, illustrate the correlation between event success and overall success for 15.1a, 15.2 and 15.3.

It's clear from the plots above that 15.2 is much more strongly correlated with success across all events than 15.1a or 15.3, but what you can also see is that the outliers in 15.3 and 15.1a are quite different.  On 15.1a, note all the dots in the top right - these are athletes who did very well on 15.1a but generally fared poorly on the rest of the workouts.  Conversely, on 15.3, note all the dots on the bottom right - these are athletes who did poorly on 15.3 but generally fared well on the rest of the workouts.

One way I've quantified this effect is to look at what I call positive outliers and negative outliers. Positive outliers are the type we saw on 15.1a (in the top left of the graph) and negative outliers are the type we saw on 15.3 (in the bottom left of the graph). In addition to looking at correlations, this metric can help us assess the type of fitness that this is.  Does this event expose a weakness (such as muscle-ups in 15.3), or does this event really allow certain athletes to shine (such as the max lift in 15.1a)?

Currently the way I'm defining a positive outlier is an athlete who finished in the top 20% of a particular workout worldwide but finished below the 50th percentile on average across the other workouts. A negative outlier is the reverse of this (bottom 20% on the workout and averaged above 50th percentile on the rest).  The chart below shows the number and percentage of outliers on each workout in 2015.  As with the correlations, this is limited to athletes who completed all 5 workouts in the Rx division.  In this case, I've shown both men's and women's outliers.

For both men and women, 15.1a had by far the most positive outliers.  Conversely, 15.3 had the most negative outliers, likely because the muscle-ups proved to be the Achilles' Heel for many an athlete. At some point soon, I'd like to go back and look at this metric for 2012-2015 to see which other events were had a large percentage of negative or positive outliers.

Now let's move on and look at a comparison of the lone repeat workout this year: 15.2.  The two charts below show the distribution of Rx scores for women in each year.  The first chart includes all athletes who completed all five events in either year, while the second chart is limited to athletes who completed all five events in both years, i.e. returning athletes.  The x-axis represents the length of time the athlete survived in the workout rather than the actual score.

Unlike what we saw in repeat workouts in 2014 and 2013, the 15.2 scores were actually better than the 14.2 scores even before limiting to just the returning athletes (you can see this by the red line being skewed to the right, with a higher % of the field surviving late into the workout).  But when we limit this to returning athletes, the disparity becomes even greater.  Nearly 70% of returning athletes got past the 3-minute mark in 15.2, compared with only about 50% in 14.2  About 35% made it past the 6-minute mark in 15.2, compared with about 20% in 14.2. Clearly, the athletes who return each year are improving.

We've focused mainly on the Rx division so far, but what impact did the scaled division have this year? Before the season, many of us figured that the addition of the scaled division played a role in the workouts that were programmed in the Rx division this year.  The charts below would seem to indicate that HQ had no choice but to add a scaled division if they wanted to program workouts like 15.3 and 15.4. The charts both show the total field at each stage of competition, split between scaled (red) and Rx (blue). The numbers represented by those bars are actually taken straight from Sam Swift's site.  But the kicker is the line on the graph, which shows what the Rx field would look like if there was no scaled division, i.e. like 2011-2014. That line represents the competitors at each stage that were "fully Rx," meaning they had not scaled at all by that point. Notice the enormous drop-off in 15.3, particularly for the women. Only about 10,000 women would have been left as of 15.5, which is fewer women than were left at the end of competition in 2013.

One thing to note is that the scoring system allowed the scaled competitors to mix in with the Rx competitors, so that some athletes who scaled a workout or two actually finished ahead of some athletes who went Rx the entire time.  If that was not allowed, the average fully Rx women's competitor would have improved their percentile ranking by 12% and the average fully Rx men's competitor would have improved their percentile ranking by 3%.  I personally don't mind the system in place now, as it incentivizes athletes to use the scaled workouts when appropriate without having to be separate from the main field.  But I'd be curious on the thoughts of others.

That wraps it up for today.  To be sure, there is more analysis to be done on the 2015 Open data, but it's time to move onto Regionals for now.  Stay tuned for a regional preview podcast in the next few weeks, as well as Regional predictions on the site as we close in on the first competitions on May 15.

Wednesday, April 8, 2015

A Look Back at the 2015 Open: Part I

We've all had time to recover from the Open, and for a (very) select few of us, it's time to move onto the regionals.  But at CFG Analysis, we're not quite finished with the Open quite yet.  On the contrary, it's time to take a thorough look back at the 2015 Open, see what the data tells us and start to understand what could be coming in the future.

Like the past two years, I'll be breaking this post up into two parts.  Today in Part I, I'll tackle the programming of this year's Open and how it compared to prior years (and what we may have expected).  Later this week in part II, I'll dig into the leaderboard a bit more to see what we can find.  If you want, see my posts (Part I and Part II) from last year for a feel for what we'll be getting into.

OK, let's get started.  Unless noted, all the metrics in this post relate to the Rx division only.

As I got started on the analysis for this post, my gut feeling was that this year's Open was a whole lot different than the past four years.  Indeed, looking at the Rx division, many of the metrics that I use to evaluate CrossFit programming were quite a bit different than in the past.  But as I looked a little closer, I found that there were really just a few key changes this year.  The biggest, in my opinion, was this:

The Open included a max-effort lift.

If you remove 15.1a (1RM clean and jerk), you're left with programming that is actually pretty similar to what we've seen before.  The chart below shows the men's loading metrics* for 2011-2015, including a version of 2015 that does not include 15.1a.  Note that without 15.1a, things look eerily similar to every other year.  With 15.1a, however, this year definitely put a larger emphasis on heavy lifting and (likely) favored larger athletes more than in the past**.

The loading metrics, of course, don't tell the whole story.  To me, there were three other key differences from prior years:
  1. Handstand push-ups appeared for the first time;
  2. The high-skill gymnastics movements (muscle-ups, handstand push-ups) appeared at the start of a workout, forcing a large portion of the field to scale;
  3. Burpees and box jumps, which accounted for about 20% of the points in the Open in previous years, did not appear at all.
Certainly #2 bothered a lot of people, but for the athletes competing for spots at regionals, this really had no effect on them.  The other two items are somewhat important, and #3 in particular was a shock to me.  As I noted on Twitter a few weeks ago, the 2015 Open was the first time since 2007 that the Open, Regionals or Games did not include burpees (excluding 2010 Regionals, when the workouts varied by region).  The chart below shows the value of each movement in the Open from 2011-2015 (including 15.1a).

Besides the big goose-eggs for burpees and box jumps, note the significance of the clean.  Since being a major player in 2011, the clean had not accounted for more than 5% of the points in an Open until this year.  The inclusion of 15.1a not only made this year a "heavier" Open, but it contributed to the Olympic-style lifts playing a huge role in the standings.  The chart below shows that the Olympic lifts and high-skill gymnastics were valued more than ever in 2015, while basic gymnastics were significantly diminished.

Although this year did include the first max-effort lift in an Open, none of the rest of the events were particularly unusual in terms of load, duration or movements.  The chart below shows the time domain (x-axis), number of movements (y-axis) and LBEL (size of ball) for each Open workout 2011-2015***.  You'll see that besides 15.1a (which I considered a 0-time domain workout despite technically lasting 6 minutes), the workouts were relatively standard compared to past Opens, though a bit on the shorter side.

It is worth noting that the 185/125 clean in 15.4 was the heaviest relative weight (1.37/0.93) ever required in an Open. That being said, I feel that the 165/110 squat clean and jerk (1.23/0.82) in 11.3 was more challenging for the community at that time than the 185-lb. clean was this year.

So where is the Open headed?  Certainly things could change in the next 10 months if there are other changes to the format (multiple scaled divisions, for instance), but it seems that the Rx division of the Open is starting to look like Regionals-lite.  I don't anticipate that the loads will get to Regional levels - the average load in metcons this year was 0.87/0.59, whereas regionals have been averaging 1.12/0.75 in the past.  However, the variety of workouts in the Open seems to be mirroring the Regionals.  I think we could see one or two of these two-part workouts each year, and I'd bet we'll have a max-effort lift each season.

I think we'll continue to see the scaled division evolve (some of the workouts were a little bland this year, in my opinion), but I think the the loads will stay similar to what we had this year, where they were about 70% of what was required for the Rx division.  Now that the community has seen the challenges in the Rx division, I'd expect more athletes to be prepared to scale early and often in the future.  There's no shame in scaling, that's for sure.

That will wrap it up for Part I of my 2015 Open recap.  Stay tuned for Part II later on this week!

*For more background on these metrics, see this post from back in 2012.

**This is something I plan to look into at some point, possibly in Part II.

***For the time-varying and load-varying workouts (including 15.1a), I took the average of the top 1,000 overall finishers from each particular year.  For instance, the top 1,000 overall male finishers took an average of 15.8 minutes on 15.2.  I also considered 15.1a and 11.3 to be single-movement workouts, despite including a clean and jerk.