Tag Archives: American Evaluation Association

Eval13: 3000+ evaluators, 800+ sessions, and way too little time to soak it all in

Last week, more than 3000 evaluators descended on my hometown of Washington, DC for the American Evaluation Association’s annual conference. I learned this much + slept this much = rockstar conference.

#omgMQP

I had the pleasure of spending Monday and Tuesday in Michael Quinn Patton’s Developmental Evaluation workshop. Due 10% to my bad vision and 90% to being starstruck, I sought out front-row seats:

Along with many other nuggets of gold, MQP shared the Mountain of Accountability, a simple visualization demonstrating a Maslow’s hierarchy for organizations. (Start with the basics like auditing, personnel review, and outputs; then progress to typical program evaluation; then progress to developmental evaluation and strategic learning.) This visual was a fan favorite; the ipads and iphones were flying around as everyone tried to snap a picture. Anyone else think that MQP would be a great addition to the dataviz TIG?

My biggest takeaway? Developmental evaluation is probably the future of evaluation, or at least the future of my evaluation career. Also, many evaluators wouldn’t call this approach “evaluation,” which means I’m not an evaluator, but an “evaluation facilitator.” Time to order new business cards!

#thumbsupviz

On Tuesday night I had Dataviz Drinks with Stephanie Evergreen, Tania Jarosewich, Andy Kirk, Johanna Morariu, Jon Schwabish, and Robert Simmon, along with a few more poor souls who had to listen to our endless enthuasiam about charts, fonts, and other things “worth staying up late for.” We’ve each been trying to reshape the dataviz community from one of frequent criticism to one of encouragement and peer learning (e.g., the Dataviz Hall of Fame.) A few beers later, the #thumbsupviz hashtag was born.

Stay tuned for our growing gallery of superb visualizations at thumbsupviz.com.

omg Factor Analysis…

On Wednesday I attended a pre-conference workshop about factor analysis. I learned the approach in grad school a few years ago, have only used it twice, and wanted to brush up my skills. The instructor provided a wealth of resources:

My biggest takeaway? Ouch. My brain was hurting. Leave the factor analysis to the experts because 99% of us are doing it wrong anyway. You don’t have to tell me twice!

Performance Management & Evaluation: Two Sides of the Same Coin

On Wednesday afternoon, I gave an Ignite presentation with my former supervisor and performance management expert, Isaac Castillo. Paired Ignites are rarely attempted, and I’m glad we took a risk. I had a lot of fun giving this talk. Stay tuned for future collaborations from Isaac and I!

Check out our slides and the recording of our presentation:

Excel Elbow Grease: How to Fool Excel into Making (Pretty Much) Any Chart You Want

On Thursday morning, I shared four strategies for making better evaluation charts: 1) adjusting default settings until your chart passes the Squint Test; 2) building two charts in one; 3) creating invisible bars; and 4) really really exploiting the default chart types, like using stacked bars to create a timeline or using a scatter plot to create a dot plot.

Here are the slides:

The section about dot plots was pretty popular, so I recorded it later:

I thought the presentation went okay, but afterwards, an audience member came up to me and asked, “So if I wanted to make a different type of chart in Excel, like anything besides a typical bar chart, how would I do it? What could I make?” “That’s what I just spent the last 45 minutes showing you.” “No I mean, if I wanted to make one of these in Excel, could I do it?” “Weren’t you in the audience for the presentation I just did?” “Yes, that would be a cool presentation, you should show us how to make those charts in Excel.” Thanks for the great idea buddy, I’ll submit that idea to next year’s conference. 🙂

East-coast happy hour

For the second year in a row, the east-coast AEA affiliates got together for a joint happy hour on Thursday night. Good vibes and familiar faces.

eval13_happy_hour

The Washington Evaluators, Baltimore Area Evaluators, New York City Consortium of Evaluators, and the Eastern Evaluation Research Society

The Conference is Over, Now What? Professional Development for Novice Evaluators

On Friday afternoon I led a roundtable with tips for novice evaluators. The discussion was awesome, especially the great chats I had with people afterwards. I’m going to write a full post recapping that session. Stay tuned!

How to Climb the R Learning Curve Without Falling Off the Cliff: Advice from Novice, Intermediate, and Advanced R Users

On Saturday morning I had the pleasure of presenting with a former teammate, Tony Fujs, and my new teammate, Will Fenn. Tony dazzled the audience with strategies for automating reports and charts with just a few lines of R code, and Will shared tips to help novices avoid falling off the learning curve cliff. Check out their resources and tips in this handout.

tony_will

Tony Fujs (left) and Will Fenn (right)

I thought the presentation went okay, but afterwards, an audience member commented, “It would be really cool if you got some evaluators together to show us what kinds of things are possible in R.” “Umm yep, that’s what we just did, Will and Tony showed how to automate reports and create data visualizations in R.” “Yep exactly, that would be a great panel, you could get several evaluators together and show how to automate reports and make data visualizations in R.” “Did you see the panel we just did?” “Yeah you should put a panel together like that.” Okay thanks, I’ll consider it. 🙂

Evaluation Blogging: Improve Your Practice, Share Your Expertise, and Strengthen Your Network

Dozens of evaluators have influenced and guided my blogging journey, and I was fortunate to co-present with three of them on Saturday: Susan Kistler, Chris Lysy, and Sheila Robinson. I first started blogging after watching Chris’ Ignite presentation at Eval11, Susan’s initial encouragement kept me going, and Sheila provides a sounding board for my new ideas.

awesome_panelists

Left to right: Susan Kistler, Chris Lysy, and Sheila B. Robinson

Can you tell we presented on Saturday morning?! Chris and I arrived early. I almost panicked, but instead Chris and I started laughing hysterically, and then a second person arrived. Close call!

empty_ballroom

By the time we started, we drew a good crowd of 30-40 bloggers and soon-to-be bloggers. Same time next year??

Evaluation Practice in the Early 21st Century

Where have we come from, and where are we headed? Evaluators have accomplished some amazing things, and the future is bright. Patrick Germain and Michelle Portlock, evaluation directors at nonprofit organizations, shared strategies for making evaluation happen when you are not in the room:

eval13_nonprofiteval

For me, the mark of a good presentation is when the evaluator shows vs. tells us something new. Kim Sabo Flores, Chad Green, Robert Shumer, David White, Javier Valdes, and Manolya Tanyu talked about incorporating youth voices into policymaking decisions. The best part: the panelists invited a youth participant to speak alongside them on the panel so that she could share her experiences firsthand.

eval13_youth_voices

They taught us about youth presence vs. participation, and then they showed us about youth presence vs. participation. Well done!

A dataviz panel shared a brief history of dataviz; strategies for displaying qualitative data; and ideas for using graphic recording:

One of many, many graphic recording examples shared by Jara Dean-Coffey

One of many, many graphic recording examples shared by Jara Dean-Coffey

The Innovation Network team is pretty fond of graphic recording too, and Kat Athanasiades even recorded an entire advocacy evaluation panel. Thanks to Cindy Banyai for capturing this awesome video!

And just in case you’re not familiar with my plans for our field…

Wave goodbye to the Dusty Shelf Report!

Wave goodbye to the Dusty Shelf Report!

Lookin’ good, Eval! See you next year in Denver!

Conference Tips for Newbie Evaluators

Wear that nametag like a rockstar

Are you getting excited for the American Evaluation Association’s conference next week in Minneapolis?

This week I’m sharing conference tips for newbie evaluators and first-time conference attendees. While many of these ideas will be obvious for experienced evaluators, these ideas won’t be obvious for your coworkers and mentees who are attending one of their first conferences. Please consider sharing these tips, along with your own advice, with the newbie evaluators in your office.

Before the Conference

Here’s what newbie evaluators and first-time conference attendees should do before an evaluation conference:

  • Assess your motivations for attending the conference.  Are you actively job-searching and want to conduct informational interviews during the conference? Are you considering starting your own consulting business and want to chat with independent consultants? Are you trying to meet others with similar interests so you can partner on future projects?
  • Start using Twitter and LinkedIn to meet people and figure out who you want to meet face-to-face at the conference.
  • Pack an extra casual outfit and an extra business outfit. Multi-day conferences tend to get more casual over time; that is, evaluators seem to wear suits the first day, then business casual for a few days, and nice jeans on the last day. Exception: If you’re presenting, save your favorite, nicest outfit for the day you’re presenting.
  • Don’t forget your electronics. It’s easy to buy a replacement toothbrush from the hotel’s convenience store, but nearly impossible to find a replacement charger for your laptop or phone.
  • Bring cash for taxis, tolls, tips, and parking meters. You’ll also need small bills ($1 and $5 bills) for group dinners. (Ever try to split a check between 20 evaluators with 20 different credit cards? It takes longer to pay the bill than eat dinner!) I usually pack $100 in small bills.
  • Find out what you’ll be reimbursed for. For meals: Does your organization use a per diem system or a reimbursement system? For alcohol: Alcohol’s typically not covered. For hotel internet access: Unless you’re working during the conference to meet a client deadline, your organization probably won’t pay that extra $10/day for internet. Instead, scope out the free wifi hotspots in the convention center or conference hotel. For airfare: Is there a limit on your airfare? Will your company reimburse you for baggage fees? For ground transportation: Will your organization reimburse you for taxis to/from the airport, or just shuttle buses? For your timesheet: Will your organization provide a charge code for the entire conference, or just the first few days? It’s not uncommon to work “on your own” on the last day of a conference, especially if it falls on a Saturday.
  • Find out which organizations will be represented at the conference. Even if you’re not job-hunting now, start brainstorming who you’ll want to work with in 5, 10, or 20 years. I like to skim through the back of the conference program.

During the Conference

Here’s what newbie evaluators and first-time conference attendees should do during an evaluation conference:

  • Iron your clothes and hang them in your closet as soon as you check-in to your hotel. Multi-day conferences are exhausting, and you’ll need all the sleep you can get over the next few days. I can’t tell you how many times I’ve overslept and ran to sessions wearing a wrinkled pair of pants…
  • Keep your conference objectives in mind. Take charge and get what you came for! Networking won’t happen unless you initiate it.
  • Don’t work during the conference, even if it means working weekends beforehand to clear your schedule. If you’re presenting, don’t work on your slides during the conference (again, even if it means working weekends beforehand to finalize your presentation). Conference time is precious and is better spent learning and talking.
  • That being said… Make time for self-care. Have quiet time in your hotel room, go to the gym, or take a walk outside.
  • Skip a session or two for networking. My most valuable lessons and tips about evaluation have come from casual conversations with other evaluators, not from formal trainings.
  • Trade business cards and jot down notes about each person to guarantee that you’ll remember them a few years from now (e.g. “Knows my friend Joe; lives in Baltimore but visits DC often; schedule coffee together.”)
  • Tweet! Use the conference’s official hashtag (#eval12 for the next American Evaluation Association conference and #eers13 for the next Eastern Evaluation Research Society conference).
  • Attend a Topical Interest Group (TIG) business meeting at the American Evaluation Association conference. This is a great way to meet current and up-and-coming leaders in the field.
  • Attend at least one session each on: 1) a brand new topic (to broaden your understanding of the field); 2) a beginner-level topic (to meet other newbies, and to feel more confident in what you already know); and 3) an advanced topic (to give yourself a reality check about what you don’t know).
  • Ask questions during sessions. Don’t doubt their project, methods, or findings, or you’ll get a reputation for being an annoying audience member. I recommend straightforward clarification questions: “This is really interesting information, thank you. I’m new to this concept/project/approach. Could you give us a little more background about xyz?”
  • Talk to presenters after their presentation. Don’t feel pressured to develop an elevator pitch or be an expert about their background or topic. Try this: “I learned a lot from your presentation, thank you. Where can I learn more about your work? Do you/your company have a website, blog, white papers, handouts, etc.? Do you ever visit DC? If so, would you like to have lunch the next time you’re in town?”
  • Don’t arrive to sessions late or leave early. It’s bad manners and distracting to the presenters.
  • Don’t worry about taking notes. Most handouts and slides will be available online right after the conference. My “notes” are my 140-character tweets.
  • Wear that nametag like a rock star.
  • And if you’re job-hunting… Print 100 copies of your resume and post them to the jobs board. Don’t share your resume with people you’re meeting for the first time unless they ask for it first.

After the Conference

Here’s what newbie evaluators and first-time conference attendees should do after an evaluation conference:

  • Connect with everyone on Twitter and with a few people on LinkedIn.
  • Send personalized, casual emails to 5-10 people you met. I’ve actually received emails with citations – “Hi Ann, I liked hearing about your work, it coincides with Smith (2010), who found a statistically significant difference between x and y. That is, until Miller (2011) employed a larger sample and found xyz….” Please don’t send anyone a literature review! Instead, write something like this: “Hi Ann, I liked hearing about your work. I’m going to apply xyz skill in my next project. Let me know if you’re ever in DC. I’d enjoy learning more over lunch sometime.”
  • Schedule coffee, lunch, and happy hours with evaluators from your city.
  • Share what you learned! Write on your personal blog, your organization’s blog, and/or the aea365 blog. Tweet. Lead a brown bag for your teammates. Bonus points: Explain what you learned to a non-evaluator (like a parent, roommate or significant other) without putting them to sleep. Translating interesting ideas from evaluationese into lay language is a skill that improves with time.

Have additional tips and tricks to share with newbie evaluators and first-time conference attendees? Please share your ideas below.

Thanks, Ann Emery

 

Additional Resources:

Defining Complex Ecologies: 1 Photo, 25 Words

As Stephanie Evergreen wrote,

“This year, the American Evaluation Association is loaded with a ton of great stuff… I’m also really excited to see the closing session on Saturday, because AEA President Rodney Hopson is pulling together something amazing.

The theme of the conference this year is Evaluation in Complex Ecologies: Relationships, Responsibilities, Relevance. What in the heck does that mean, right? Well, this is your chance to help define it. Take a picture, wrap it in 25 words, and send it to AEA2012closingphotos@gmail.com before noon CT on Thursday, October 25. The best photos will be chosen and featured in the closing session on Saturday October 27, 2012 at 4:30 pm. Dudes, there will be prizes.

If it isn’t obvious to you already, what’s so so cool about this project is that it is an attempt to let people get visual with concepts that are pretty darn abstract. We, serious and academic researchers and evaluators, often shy away from exploring a bit of creativity. This is an officially sanctioned license to go play (even for those of you who aren’t headed to the conference this year – you can still participate).”

Although I was tempted to submit something like this, here’s my submission:

Internal evaluator seeks stakeholder buy-in and learning culture. Willing to wear hairnets, plastic gloves, and serve tortilla chips to 500+ children during organization’s Thanksgiving dinner.

And the countdown begins: 19 days until #eval12

When I realized that the American Evaluation Association’s conference is only 19 days away:

 

When I checked the schedule and realized the sessions start at 7am:

 

When I’m already flooded with work before the conference:

 

When I still have to finish a few presentations before the conference:

 

When I thought, hey, it’s not so bad, because this is how I felt at my first evaluation conference:

When I remembered there would be be plenty of happy hours at the conference:

 

And when I remembered I’d get to see all my favorite evaluators in one place:

The Volume of Your Presentation [Guest post by Isaac Castillo]

I’m getting excited for the American Evaluation Association’s Potent Presentations Initiative because it will offer training on messaging, design, and delivery to help evaluators transform into rockstar presenters. Although many of us are beginners at delivering presentations, others are veterans or experts. Today I’ve invited one of my favorite expert-level presenters, Isaac Castillo, to share some of his presentation tips with us. Isaac is also a great resource about performance management and youth development, and you can follow him on Twitter here: @isaac_outcomes.

Enjoy! Ann Emery

——————————–

Isaac Castillo

“One of the most difficult things for many presenters to master is volume control of their voice.   In most situations, we are used to talking in a normal conversational tone and volume.  But normal volume usually doesn’t work for a presentation – and it certainly doesn’t work if you are presenting without a microphone.   On the other hand, you don’t want to be screaming at your audience, particularly in a small space.

So how do you know if your volume is correct?  Here are some tips to keep in mind.

If at all possible, get to your presentation space early and ask someone to help check your volume.  This could be a friend or colleague, the person responsible for audio/visual needs, or even that eager person that shows up first.  Keep in mind that what sounds loud enough at the beginning of your presentation in an empty room will NOT be loud enough when dozens (or hundreds) of people fill the room.

But sometimes you won’t have the chance to check your volume in a room before you start.  In these cases, here are some things you can do to make sure everyone can hear you.

First, you can simply ask your audience if you are loud enough.  There is nothing wrong with this, and really is one of the best ways to check your volume.  The key is to make sure you get feedback from the back of the room – those in the front row will always be able to hear you, you want to make sure the same is true for those in the last row.

Second, you should look for signs that members of the audience are having difficulty hearing you.  Looks of confusion, people leaning forward, and tilted heads are all good signs that you are not being loud enough.  And again, pay attention to those in the back row.

Tips for Beginners:   Talk slightly louder than you think you should be talking.  If you feel like you are talking in a normal voice, you are not loud enough.  You should really feel like you are talking slightly louder than necessary – that will be your correct volume for a filled room.

Tips for Veterans:  Work out your diaphragm!  Your diaphragm is where you get your vocal volume.   Do this exercise to increase your capacity to talk louder:  lie on your back and place a book (nothing too heavy) on your stomach directly below your ribcage.   Take a deep breath in and lift the book by inflating your diaphragm.  Breathe out and let the book fall with your stomach.  Practice this for five minutes a day – increasing the weight of the books.  You should eventually be able to lift two telephone books with no effort.

Tips for Experts:   Use different volume levels during your presentation.   Telling stories or communicating content at different volume levels can add a different level of emphasis to your presentation.  I like to use a softer voice when talking about difficult work or when telling emotional stories.  Louder voices are good for exciting moments.  Varying your volume intentionally within your presentation can help make a good presentation great.”

–Isaac Castillo