Eval13: 3000+ evaluators, 800+ sessions, and way too little time to soak it all in

Last week, more than 3000 evaluators descended on my hometown of Washington, DC for the American Evaluation Association’s annual conference. I learned this much + slept this much = rockstar conference.


I had the pleasure of spending Monday and Tuesday in Michael Quinn Patton’s Developmental Evaluation workshop. Due 10% to my bad vision and 90% to being starstruck, I sought out front-row seats:

Along with many other nuggets of gold, MQP shared the Mountain of Accountability, a simple visualization demonstrating a Maslow’s hierarchy for organizations. (Start with the basics like auditing, personnel review, and outputs; then progress to typical program evaluation; then progress to developmental evaluation and strategic learning.) This visual was a fan favorite; the ipads and iphones were flying around as everyone tried to snap a picture. Anyone else think that MQP would be a great addition to the dataviz TIG?

My biggest takeaway? Developmental evaluation is probably the future of evaluation, or at least the future of my evaluation career. Also, many evaluators wouldn’t call this approach “evaluation,” which means I’m not an evaluator, but an “evaluation facilitator.” Time to order new business cards!


On Tuesday night I had Dataviz Drinks with Stephanie Evergreen, Tania Jarosewich, Andy Kirk, Johanna Morariu, Jon Schwabish, and Robert Simmon, along with a few more poor souls who had to listen to our endless enthuasiam about charts, fonts, and other things “worth staying up late for.” We’ve each been trying to reshape the dataviz community from one of frequent criticism to one of encouragement and peer learning (e.g., the Dataviz Hall of Fame.) A few beers later, the #thumbsupviz hashtag was born.

Stay tuned for our growing gallery of superb visualizations at thumbsupviz.com.

omg Factor Analysis…

On Wednesday I attended a pre-conference workshop about factor analysis. I learned the approach in grad school a few years ago, have only used it twice, and wanted to brush up my skills. The instructor provided a wealth of resources:

My biggest takeaway? Ouch. My brain was hurting. Leave the factor analysis to the experts because 99% of us are doing it wrong anyway. You don’t have to tell me twice!

Performance Management & Evaluation: Two Sides of the Same Coin

On Wednesday afternoon, I gave an Ignite presentation with my former supervisor and performance management expert, Isaac Castillo. Paired Ignites are rarely attempted, and I’m glad we took a risk. I had a lot of fun giving this talk. Stay tuned for future collaborations from Isaac and I!

Check out our slides and the recording of our presentation:

Excel Elbow Grease: How to Fool Excel into Making (Pretty Much) Any Chart You Want

On Thursday morning, I shared four strategies for making better evaluation charts: 1) adjusting default settings until your chart passes the Squint Test; 2) building two charts in one; 3) creating invisible bars; and 4) really really exploiting the default chart types, like using stacked bars to create a timeline or using a scatter plot to create a dot plot.

Here are the slides:

The section about dot plots was pretty popular, so I recorded it later:

I thought the presentation went okay, but afterwards, an audience member came up to me and asked, “So if I wanted to make a different type of chart in Excel, like anything besides a typical bar chart, how would I do it? What could I make?” “That’s what I just spent the last 45 minutes showing you.” “No I mean, if I wanted to make one of these in Excel, could I do it?” “Weren’t you in the audience for the presentation I just did?” “Yes, that would be a cool presentation, you should show us how to make those charts in Excel.” Thanks for the great idea buddy, I’ll submit that idea to next year’s conference. 🙂

East-coast happy hour

For the second year in a row, the east-coast AEA affiliates got together for a joint happy hour on Thursday night. Good vibes and familiar faces.


The Washington Evaluators, Baltimore Area Evaluators, New York City Consortium of Evaluators, and the Eastern Evaluation Research Society

The Conference is Over, Now What? Professional Development for Novice Evaluators

On Friday afternoon I led a roundtable with tips for novice evaluators. The discussion was awesome, especially the great chats I had with people afterwards. I’m going to write a full post recapping that session. Stay tuned!

How to Climb the R Learning Curve Without Falling Off the Cliff: Advice from Novice, Intermediate, and Advanced R Users

On Saturday morning I had the pleasure of presenting with a former teammate, Tony Fujs, and my new teammate, Will Fenn. Tony dazzled the audience with strategies for automating reports and charts with just a few lines of R code, and Will shared tips to help novices avoid falling off the learning curve cliff. Check out their resources and tips in this handout.


Tony Fujs (left) and Will Fenn (right)

I thought the presentation went okay, but afterwards, an audience member commented, “It would be really cool if you got some evaluators together to show us what kinds of things are possible in R.” “Umm yep, that’s what we just did, Will and Tony showed how to automate reports and create data visualizations in R.” “Yep exactly, that would be a great panel, you could get several evaluators together and show how to automate reports and make data visualizations in R.” “Did you see the panel we just did?” “Yeah you should put a panel together like that.” Okay thanks, I’ll consider it. 🙂

Evaluation Blogging: Improve Your Practice, Share Your Expertise, and Strengthen Your Network

Dozens of evaluators have influenced and guided my blogging journey, and I was fortunate to co-present with three of them on Saturday: Susan Kistler, Chris Lysy, and Sheila Robinson. I first started blogging after watching Chris’ Ignite presentation at Eval11, Susan’s initial encouragement kept me going, and Sheila provides a sounding board for my new ideas.


Left to right: Susan Kistler, Chris Lysy, and Sheila B. Robinson

Can you tell we presented on Saturday morning?! Chris and I arrived early. I almost panicked, but instead Chris and I started laughing hysterically, and then a second person arrived. Close call!


By the time we started, we drew a good crowd of 30-40 bloggers and soon-to-be bloggers. Same time next year??

Evaluation Practice in the Early 21st Century

Where have we come from, and where are we headed? Evaluators have accomplished some amazing things, and the future is bright. Patrick Germain and Michelle Portlock, evaluation directors at nonprofit organizations, shared strategies for making evaluation happen when you are not in the room:


For me, the mark of a good presentation is when the evaluator shows vs. tells us something new. Kim Sabo Flores, Chad Green, Robert Shumer, David White, Javier Valdes, and Manolya Tanyu talked about incorporating youth voices into policymaking decisions. The best part: the panelists invited a youth participant to speak alongside them on the panel so that she could share her experiences firsthand.


They taught us about youth presence vs. participation, and then they showed us about youth presence vs. participation. Well done!

A dataviz panel shared a brief history of dataviz; strategies for displaying qualitative data; and ideas for using graphic recording:

One of many, many graphic recording examples shared by Jara Dean-Coffey

One of many, many graphic recording examples shared by Jara Dean-Coffey

The Innovation Network team is pretty fond of graphic recording too, and Kat Athanasiades even recorded an entire advocacy evaluation panel. Thanks to Cindy Banyai for capturing this awesome video!

And just in case you’re not familiar with my plans for our field…

Wave goodbye to the Dusty Shelf Report!

Wave goodbye to the Dusty Shelf Report!

Lookin’ good, Eval! See you next year in Denver!

21 thoughts on “Eval13: 3000+ evaluators, 800+ sessions, and way too little time to soak it all in

  1. Sheila B Robinson, Ed. D

    Love it! You definitely get a #thumbsupviz for this: I learned this much + slept this much = rockstar conference. I can’t reproduce your boldface and grey text here, but when I read it, it struck me as a fine example of qualitative data visualization. 🙂 I so enjoyed presenting with you (was I the only one who noticed that we sat down on the stage in the same order as we appear were on the handout?) and hope to have another opportunity next year. I am always in awe of your creativity (as well as your incredible eval know-how) and propose a new hashtag: #omgake. Yup, I predict as part of the future of eval, that one will be trending VERY soon!

    1. Ann K. Emery Post author

      #omgSBR! Thanks. I had a great time presenting with you, too. I was pleasantly surprised at the interest in blogging, and I think it’s only a matter of time until we see more and more evaluators joining the online community. (P.S. Do I have to wait until Denver to see you again?! Let me know if you’re ever in DC or MD.)

    1. Ann K. Emery Post author

      Thanks Mary! I enjoyed reading your conference recap. I linked to your top 10 highlights in a comment (below). Can’t wait to read more from your new blog! Keep up the good work.

  2. Ann K. Emery Post author

    Hey all, check out these awesome conference recaps:

    – Mary Nash: http://marysnash.com/2013/10/20/reflections-on-my-first-aea-conference/
    – aea365 Twitter users: http://aea365.org/blog/?p=10362
    – Johanna Morariu’s #eval13 social network map: https://twitter.com/j_morariu/status/393119964334288897
    – Chris Lysy, Day 1: http://freshspectrum.com/eval13-day-one/
    – Chris Lysy, Day 2: http://freshspectrum.com/eval13-day-two/
    – Chris Lysy, Day 3: http://freshspectrum.com/eval13-day-three/
    – Chris Lysy, Day 4: http://evalcentral.com/2013/10/19/eval13-day-four/
    – David Akana: http://storify.com/akanadavid/recap-of-2013-aea-conference

    Have you seen additional conference recaps? Please share links! Thanks.

    1. Ann K. Emery Post author

      A few more excellent conference reflections:

      – Chi Yan Lam on Tom Chapel’s logic model workshop: http://chiyanlam.com/2013/10/15/key-takeaways-from-tom-chapels-aea13-workshop-logic-models-for-program-evaluation-and-planning/
      – Chi Yan Lam on Michael Quinn Patton’s State of Developmental Evaluation session: http://chiyanlam.com/2013/10/17/highlights-from-michael-quinn-pattons-eval13-talk-on-the-state-of-developmental-evaluation/
      – Jane Davidson about the Case Studies of Evaluators’ Lives panel: http://genuineevaluation.com/case-studies-of-evaluators-lives-a-cultural-perspective-yes-culture/
      – John Gargani about Day 1: http://evalblog.com/2013/10/16/on-the-ground-at-aea-1-tina-and-rodney/

    2. Ann K. Emery Post author

      And some more:

      – James Pann’s conference recap, on his brand new evaluation blog: http://jamespann.net/?p=19
      – Ann Price’s eval13 musings: http://www.communityevaluationsolutions.com/eval-2013-musings/
      – JSI Inc.’s awesome photo booth about the meaning of evaluation in the 21st century: https://www.facebook.com/media/set/?set=a.10151707172123016.1073741829.87117168015&type=3&uploaded=68
      – Bronwyn Mauldin’s recap of her dataviz session: http://tumblacre.tumblr.com/post/64580240708/an-ethnographic-study-of-data-visualizers

    3. Ann K. Emery Post author

      And the latest conference recaps include:

      – Sheila B. Robinson’s full recap: http://sheilabrobinson.com/2013/10/25/eval13-omgmqp-esm-dataviz-program-design-blogging-and-the-great-big-nerd-project/
      – Sheila B. Robinson’s aea365 post: http://aea365.org/blog/?p=10381
      – Bailey Murph’s experiences as a GEDI at her first AEA conference: http://planningcouncil.wordpress.com/2013/10/29/a-gedi-goes-to-washington/
      – Washington University’s Center for Public Health Systems Science: http://cphss.wustl.edu/NewsAndEvents/Pages/AEA-2013.aspx
      – Tom Kelly’s first (http://tomeval.com/?p=76) and second (http://tomeval.com/?p=81) blog posts, which reference AEA learnings

    4. Ann K. Emery Post author

      Conference recaps from the past week include:

      – DME for Peace: http://www.dmeforpeace.org/discuss/reflections-american-evaluation-association-aea-conference
      – John Gargani and “post-parting depression:” http://evalblog.com/2013/10/30/on-the-ground-at-aea-2-what-participants-had-to-say/
      – Chris Lysy’s reflections: http://freshspectrum.com/eval13-reflections/
      – Trina Willard’s top 10 conference highlights: http://knowledgeadvisorygroup.com/the-top-10-reasons-why-i-loved-attending-evaluation-2013/
      – Innovation Network’s newsletter recap: http://myemail.constantcontact.com/Conference-Recap–Evaluation-Practice-in-the-Early-21st-Century.html?soid=1011133167174&aid=wNPzXW4NRUY
      – Johanna Morariu’s guest blog post with an #eval13 social network map: http://emeryevaluation.com/2013/11/01/morariu/
      – Tom Archibald’s first post on his brand new blog! http://tgarchibald.wordpress.com/2013/11/04/on-the-origins-of-this-blog/

      Am I missing any? Thanks! Ann

  3. Michael Quinn Patton

    Ann, Welcome to the world of evaluation facilitation, which you will clearly help create well into the 21st Century. Thanks for your enthusiasm, insights, commitment and great sense of humor. You and your fellow tweeters (though you appear to be the champ) make me optimistic about evaluation’s future. MQP

  4. Karen

    Great post Ann. The Mountain of Accountability really caught my eye. Know how I can get a copy…? I was just emailing my GEDI…Graduate Education Diversity Internship lol colleagues about how missing the conference this year was like missing a family reunion…sorta. I love the conference energy and hanging out with old and new eval friends. Speaking of new friends, ever thought about inviting the Atlanta Area Evaluation Association to your affiliate gathering? #EastCoastAffiliates gotta stick together. We’re a fun bunch, promise! #teamECA

    Kudos to you and all of your achievements!


    1. Ann K. Emery Post author

      Karen, You were greatly missed at the conference!

      I don’t think the Mountain of Accountability is available online yet, but I’m guessing we’ll see more of it within the next year or so.

      Did you see the recent conference recap from Bailey Murph, a current GEDI? http://planningcouncil.wordpress.com/2013/10/29/a-gedi-goes-to-washington/

      And consider this your official invite to next year’s affiliate happy hour! But beware… you’ve just volunteered yourself to help with the heavy lifting, like choosing a bar within walking distance of the hotel and sending the evite to your affiliate members. 🙂

  5. Trina Willard

    Ann – what a terrific spin-down of all things Evaluation 2013! Thanks so much for being such a wonderful and creative contributor to the evaluation community so we can all keep learning!

  6. Nicole Clark

    This is such a great post! Thanks so much! I hate that I missed Evaluation 2013, but I hope to be in attendance in 2014!

  7. Laura Keene

    Thanks for another great post! I’m very jealous of the personal MQP message you received. ; ) Quick random question…which software/tool did you use to create the dot plots video? I need to create something just like that (audio + sharing my screen) and would greatly appreciate any recommendations you have. Thanks again!!

    1. Ann K. Emery Post author

      Thanks Laura!

      To make the dot plots video, I: 1) opened up my powerpoint slides, 2) opened up http://www.screenr.com, 3) pressed record, 4) talked through the slides just as I had during my live conference presentation, and 5) uploaded the video to YouTube and embedded it here on my blog.

      I use Screenr for all my recordings at http://emeryevaluation.com/excel/ and http://www.youtube.com/annkemery. It’s free and extremely user-friendly. It only gives you 5 minutes of recording time, so you have to stay on track, no rambling allowed. (However, I track my YouTube analytics pretty closely, and 2-3 minutes is the magic length for videos – a very low percentage of viewers will watch an entire 4 or 5 minute video).

      If you really really want longer videos, other options include Jing and Camtasia. This year’s eval13 Ignite sessions were edited using Camtasia; you can check out AEA’s YouTube channel at http://www.youtube.com/amevalassn to see the cool features.

      Best of luck with your recordings. Please share a link when you’re finished so I can see it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s