Category Archives: Tips for Novice Evaluators

The Conference is Over, Now What? Professional Development for Novice Evaluators

At the American Evaluation Association’s annual conference in October 2013, I led a roundtable titled “The Conference is Over, Now What? Professional Development for Novice Evaluators.” We discussed ways that novices can deepen their knowledge, build their skills, socialize with other evaluators, and get involved in leadership positions. I compiled the notes here so more people can benefit from these resources.

Here are the best resources for novice evaluators:

aea365

aea365This is the American Evaluation Association’s daily blog located at aea365.org. You can read about everything from item response theory to slide design. Confession: I rarely read an entire post. Instead, I’m skimming the posts just to see the title, author, author’s organization, and the main gist of the content. This is a great way to stay up-to-date on the biggest trends in the field.

You should seriously write for aea365, probably 2-3 times a year, even if you’re new to the field. Just make sure you follow the contribution guidelines.

Affiliates and other organizations

AEA is the national-level mothership and there are more than 20 local and regional affiliates. You can find a full listing of affiliates here: http://www.eval.org/p/cm/ld/fid=12. Every affiliate is different. For example, the Washington Evaluators hold monthly brown bags, quarterly happy hours, and an annual holiday party. The Eastern Evaluation Research Society holds an annual 3-day conference. Other affiliates hold virtual book clubs, maintain blogs, or simply hold member meetings via teleconference.

You should join your affiliate. Seriously. The mailing lists are little nuggets of gold and worth every penny of that $25/year membership. The Washington Evaluators, for example, send job announcements almost every day, so you’ll always know which organizations are hiring and expanding. Don’t forget to attend the affiliate events too. (Sometimes people just pay dues but skip all the events, and then they don’t know why they’re not meeting anyone? This confuses me.) After a year, start planning small events yourself, like a brown bag. Then, join the Board.

Here are some additional reasons to join affiliates and ideas for getting involved:

data_community_dcThere are tons of additional evaluation groups. For example, the Environmental Evaluators Network, led by Matt Keene, holds forums for evaluators interested in environmental issues. If you’re in Washington, DC, the Aspen Institute holds quarterly breakfast panels focused on advocacy evaluation. At Innovation Network, we hold Ask an Evaluator sessions for nonprofit leaders. Tony Fujs and I also attend Data Science DCData Visualization DC, and Data Community DC monthly meetups. No matter your city, there are probably lots of events that fit your interests.

Blogs

First, check out evalcentral.com, run by Chris Lysy. Chris pulls in feeds from 60+ evaluation blogs so you’ll get exposed to a diverse set of perspectives. Chris even developed a daily email digest, so you can subscribe once to all 60+ blogs rather than monitoring your subscriptions to all the individual blogs. I suggest setting EvalCentral as one of your homepage tabs (along with your other must-haves like Gmail and Pandora) so it’s there every time you log into your computer. And again, I rarely read an entire blog post but I skim everything for the title, author, and main gist of what they’re talking about.

Second, check out AEA’s listing of evaluators and evaluation organizations who blog: http://www.eval.org/p/cm/ld/fid=71

I started blogging after watching Chris Lysy’s Ignite presentation at the 2011 AEA conference. Here’s Chris’ Ignite, which outlines just a few of the infinite reasons why evaluators should blog:

Coffee Break webinars

Coffee Break webinars are just 20 minutes long, so they’re a perfect way to squeeze in some quick professional development in the middle of a busy work day. The best part? They’re free for AEA members. I like to sign up for topics that I know nothing about. After 20 minutes, I’m not an expert, but at least I’ve got a basic understanding of that flavor of evaluation.

Conferences

Evaluation conferences include:

Do you know of additional evaluation conferences? Please link to them in the comments section below.

I also like to attend non-evaluation conferences to hear how non-evaluators are describing our work (they have completely different lingo and tend to value qualitative data way more than evaluators do).

eStudies

An eStudy is a 3- to 6-hour webinar run by AEA. eStudies are like mini grad school courses because they go in-depth on a particular topic (as opposed to 20-minute Coffee Break webinars, which just provide an overview of a topic). eStudies are broken into 90-minute chunks and there’s typically a homework assignment between each segment to help you practice your new skills.

For example, I participated in an eStudy about nonparametric statistics in which the instructor covered about 20 different nonparametric statistics, when to use each one, and how to perform the calculations in SPSS. We even got to keep her slides, which were full of step-by-step SPSS screenshots. Almost two years later, I still pull out my eStudy notes whenever I need to use some nonparametric statistics.

Journals

AEA offers two journals, the American Journal of Evaluation and New Directions for Evaluation. Both of these journals are included with your AEA membership. What a steal!

LinkedIn

These days, I can’t imagine an employer not doing a full internet search on new applicants. Make sure your LinkedIn profile has, at the bare minimum, a professional photo, your full work history (including dates), and your education history. You can also use LinkedIn to build your online portfolio (e.g., embedded slideshows from recent conference presentations, links to publications and projects, and your list of certifications).

Want to connect with other evaluators? Some awesome evaluation groups on LinkedIn include:

Do you know of additional evaluation groups on LinkedIn? Share your suggestion in the comments below. Thanks!

Listservs, mailing lists, and newsletters

First, check out EvalTalk: https://listserv.ua.edu/archives/evaltalk.html. This is a traditional listserv that goes directly to your email inbox. Subscribing to EvalTalk is a must (if only to watch the bloodbath as evaluators battle each other online). Make sure you adjust your settings so that you get a daily or weekly digest – otherwise you’ll drown in the sheer volume of messages.

Second, subscribe to mailing lists and newsletters specific to your client projects. Whenever I begin a new project, I search the client’s website and subscribe to everything I can (like their Twitter feed, email newsletter, and blog). As a consultant, I only see one slice of their work. Subscribing to all of their updates helps me get a fuller picture of their work, so I can make sure the evaluation fits their organization’s culture and needs.

Thought Leaders Discussion Series

AEA’s Thought Leaders Discussion Series is like a big message board to debate bigger-picture, theoretical issues in the field. Each series is led by a different person and has a different flavor.

Topical Interest Groups (TIGs)

Topical Interest Groups (TIGs) are known as affinity groups in other professional associations. You get to select five TIGs when you join AEA, and you can change your selection at any time. Each TIG is different–different sizes, leadership and committee structure, and different business meetings. I suggest attending business meetings for multiple TIGs at each conference. See which culture fits you best. After a few years, get more involved by running for a leadership position.

Twitter

Just getting started on Twitter? Here’s my list of 275+ evaluators and 80+ evaluation organizations who are using Twitter. Use #eval13 to tweet about that year’s AEA conference (not #AEA13 – the poor folks at the American Equine Association will get confused). Use #eval for all your regular evaluation-related content.

There's a huge online evaluation community. What are you waiting for?!

Here’s Johanna Morariu’s social network map of the #eval13 hashtag. There’s a huge online evaluation community. What are you waiting for?!

White papers and other gray literature

There are approximately 8000 evaluators in the American Evaluation Association. I estimate that maybe… 5%?… aim to publish articles in academic journals. Most of us are practitioners and consultants (not academics, theorists, or professors). White papers and other gray literature are a great way to learn about our work, our insights, and our tips. For examples, check out innonet.org/research and evaluationinnovation.org/publications.

Additional resources

What are your favorite resources? Which resources were most valuable during your first few years in the field? And, most importantly–do you have different viewpoints on any of the resources I described? Share your perspectives! I’ve presented one opinion and there are many more to add to the mix.

Learning Evaluation through Apprenticeships [Evaluation musings by Cindy Tananis]

cindy_tananis

Cindy Tananis, Director, Collaborative for Evaluation and Assessment Capacity at the University of Pittsburgh

Greetings! I’m Cindy Tananis. The folks at the University of Pittsburgh’s Collaborative for Evaluation and Assessment Capacity (CEAC) have been thinking about blogging for some time now — and we’ve decided that if EVERYONE starts their OWN blog, then its hard to talk ACROSS blogs, so — we’ve decided to add some of our thinking here with Ann’s blog!

First. Let me do something that we think is really important in evaluation: CONTEXTUALIZE!  Before you can really understand findings from evaluation, you have to understand context.  Here’s ours!  We are a university based (University of Pittsburgh), evaluation group (of faculty and doctoral students in the School of Education) that works with folks who are trying to make a difference in the world of learning for children and adults.  Our work spans lots of ranges — small to large scale, quant to qual focus, formal to informal settings, school based to community based — and more.  I am the founder and Director of CEAC and also am an Associate Professor in Administrative and Policy Studies at Pitt, specializing in evaluation and educational leadership studies.  All of our evaluators are doc students studying some aspect of education, though that too represents a broad range of interests.

Now — on to the more typical bloglike entry!

It probably makes sense, as an introduction to think a bit about LEARNING EVALUATION.  We do that essentially through APPRENTICESHIP.  Doc students come to CEAC to fund their doctoral studies — and those studies may be in K-12 or Higher Ed leadership or in Social and Comparative Analysis of Ed — or Research Methods — or other specialized areas.  We do offer some evaluation and methods courses, but we don’t currently offer an evaluation area of study or degree option.  So, our folks come to us with very little evaluation expertise or experience — a perfect place for a ground up apprenticeship model.  Here are a few things we’ve learned along the way over the last six years that CEAC has been in operation:

  1. Learning by doing, works.  Folks can learn the basic skills as well as more nuanced practice of evaluation through practical work.  Reading and studying some of the seminal texts and theorists in the field adds to that, especially for a more conceptual and theoretical grounding for the work.
  2. People learn better by working together.  We do ALL of work as teams — this allows someone who even has a week more experience with an aspect of a project to become the “teacher” as others are students — with a fluid movement across roles that may flip flop even numerous times through a day!  We really do connect with the ideas of a community of practice from the literature.
  3. Learning is job one for everyone.  The fact that we are situated in a university, and better yet, a school of education, helps to make this point, but I think it resonates across all evaluation.  Evaluation helps people learn what they need to know to make sense of phenomena — typically to make decisions related to worth and value (though not always).  So, focusing on learning keeps us all good evaluation learners — that makes us good evaluators, and we hope, for some of us, better evaluation thinkers/theorists.
  4. Essentially, we are all apprentices in a field that is diverse and fluid.   One of the characteristics that first had me fall in love with the field is the diversity of experience, focus, theory, and practical application represented in evaluation.  As Scriven instructs us, evaluation is a trans-discipline — it informs and influences many other disciplines.  That provides us with a great opportunity to learn across fields, apply our work in new settings, work with partners from other disciplines —- its an invitation to  constantly challenge our knowledge and reform it anew.

CEAC_logoWant to learn more? Stop by and visit our website to learn more about CEAC:  www.ceac.pitt.edu

Top Tips for the Evaluators of Tomorrow

guardian_top_tipsDear evaluators,

‘Tis the season for giving thanks. I’m thankful for the experienced evaluators who have guided me, mentored me, and nudged me in the right direction over the past few years. Thank you!

A few months ago, I discovered the 9 top tips for the journalists of tomorrow by the Guardian. They collected tips about the tools and skills that novice journalists need to survive — and thrive — in journalism. What if we created a similar guide for evaluation?

I’d like to collect and compile your career advice for novice evaluators. Want to help me pay it forward? Just add a “comment” below on one or more of the following topics:

  • What’s the best evaluation advice you’ve received? Who was it from? How did it affect your everyday evaluation practice?
  • What do you wish you knew about evaluation earlier? (Check out Karen Anderson’s take on this question here.)
  • What are your interview, resume, and cover letter tips for job candidates? (Or, your pet peeves that candidates should avoid?)
  • What are the must-have technical skills and must-have interpersonal traits for the next generation of evaluators?
  • “__________ has been the most challenging aspect of evaluation for me personally, and __________ is how I deal with it.”
  • “__________ will be the biggest challenge facing our field over the next 10-20 years, and __________ is how we might deal with it.” (Check out more predictions about the future of evaluation here.)

Thanks for your commitment to our field.

Ann

P.S. Thanks to everyone who’s shared their tips via Twitter! Here are a few of those comments:

Conference Tips for Newbie Evaluators

Wear that nametag like a rockstar

Are you getting excited for the American Evaluation Association’s conference next week in Minneapolis?

This week I’m sharing conference tips for newbie evaluators and first-time conference attendees. While many of these ideas will be obvious for experienced evaluators, these ideas won’t be obvious for your coworkers and mentees who are attending one of their first conferences. Please consider sharing these tips, along with your own advice, with the newbie evaluators in your office.

Before the Conference

Here’s what newbie evaluators and first-time conference attendees should do before an evaluation conference:

  • Assess your motivations for attending the conference.  Are you actively job-searching and want to conduct informational interviews during the conference? Are you considering starting your own consulting business and want to chat with independent consultants? Are you trying to meet others with similar interests so you can partner on future projects?
  • Start using Twitter and LinkedIn to meet people and figure out who you want to meet face-to-face at the conference.
  • Pack an extra casual outfit and an extra business outfit. Multi-day conferences tend to get more casual over time; that is, evaluators seem to wear suits the first day, then business casual for a few days, and nice jeans on the last day. Exception: If you’re presenting, save your favorite, nicest outfit for the day you’re presenting.
  • Don’t forget your electronics. It’s easy to buy a replacement toothbrush from the hotel’s convenience store, but nearly impossible to find a replacement charger for your laptop or phone.
  • Bring cash for taxis, tolls, tips, and parking meters. You’ll also need small bills ($1 and $5 bills) for group dinners. (Ever try to split a check between 20 evaluators with 20 different credit cards? It takes longer to pay the bill than eat dinner!) I usually pack $100 in small bills.
  • Find out what you’ll be reimbursed for. For meals: Does your organization use a per diem system or a reimbursement system? For alcohol: Alcohol’s typically not covered. For hotel internet access: Unless you’re working during the conference to meet a client deadline, your organization probably won’t pay that extra $10/day for internet. Instead, scope out the free wifi hotspots in the convention center or conference hotel. For airfare: Is there a limit on your airfare? Will your company reimburse you for baggage fees? For ground transportation: Will your organization reimburse you for taxis to/from the airport, or just shuttle buses? For your timesheet: Will your organization provide a charge code for the entire conference, or just the first few days? It’s not uncommon to work “on your own” on the last day of a conference, especially if it falls on a Saturday.
  • Find out which organizations will be represented at the conference. Even if you’re not job-hunting now, start brainstorming who you’ll want to work with in 5, 10, or 20 years. I like to skim through the back of the conference program.

During the Conference

Here’s what newbie evaluators and first-time conference attendees should do during an evaluation conference:

  • Iron your clothes and hang them in your closet as soon as you check-in to your hotel. Multi-day conferences are exhausting, and you’ll need all the sleep you can get over the next few days. I can’t tell you how many times I’ve overslept and ran to sessions wearing a wrinkled pair of pants…
  • Keep your conference objectives in mind. Take charge and get what you came for! Networking won’t happen unless you initiate it.
  • Don’t work during the conference, even if it means working weekends beforehand to clear your schedule. If you’re presenting, don’t work on your slides during the conference (again, even if it means working weekends beforehand to finalize your presentation). Conference time is precious and is better spent learning and talking.
  • That being said… Make time for self-care. Have quiet time in your hotel room, go to the gym, or take a walk outside.
  • Skip a session or two for networking. My most valuable lessons and tips about evaluation have come from casual conversations with other evaluators, not from formal trainings.
  • Trade business cards and jot down notes about each person to guarantee that you’ll remember them a few years from now (e.g. “Knows my friend Joe; lives in Baltimore but visits DC often; schedule coffee together.”)
  • Tweet! Use the conference’s official hashtag (#eval12 for the next American Evaluation Association conference and #eers13 for the next Eastern Evaluation Research Society conference).
  • Attend a Topical Interest Group (TIG) business meeting at the American Evaluation Association conference. This is a great way to meet current and up-and-coming leaders in the field.
  • Attend at least one session each on: 1) a brand new topic (to broaden your understanding of the field); 2) a beginner-level topic (to meet other newbies, and to feel more confident in what you already know); and 3) an advanced topic (to give yourself a reality check about what you don’t know).
  • Ask questions during sessions. Don’t doubt their project, methods, or findings, or you’ll get a reputation for being an annoying audience member. I recommend straightforward clarification questions: “This is really interesting information, thank you. I’m new to this concept/project/approach. Could you give us a little more background about xyz?”
  • Talk to presenters after their presentation. Don’t feel pressured to develop an elevator pitch or be an expert about their background or topic. Try this: “I learned a lot from your presentation, thank you. Where can I learn more about your work? Do you/your company have a website, blog, white papers, handouts, etc.? Do you ever visit DC? If so, would you like to have lunch the next time you’re in town?”
  • Don’t arrive to sessions late or leave early. It’s bad manners and distracting to the presenters.
  • Don’t worry about taking notes. Most handouts and slides will be available online right after the conference. My “notes” are my 140-character tweets.
  • Wear that nametag like a rock star.
  • And if you’re job-hunting… Print 100 copies of your resume and post them to the jobs board. Don’t share your resume with people you’re meeting for the first time unless they ask for it first.

After the Conference

Here’s what newbie evaluators and first-time conference attendees should do after an evaluation conference:

  • Connect with everyone on Twitter and with a few people on LinkedIn.
  • Send personalized, casual emails to 5-10 people you met. I’ve actually received emails with citations – “Hi Ann, I liked hearing about your work, it coincides with Smith (2010), who found a statistically significant difference between x and y. That is, until Miller (2011) employed a larger sample and found xyz….” Please don’t send anyone a literature review! Instead, write something like this: “Hi Ann, I liked hearing about your work. I’m going to apply xyz skill in my next project. Let me know if you’re ever in DC. I’d enjoy learning more over lunch sometime.”
  • Schedule coffee, lunch, and happy hours with evaluators from your city.
  • Share what you learned! Write on your personal blog, your organization’s blog, and/or the aea365 blog. Tweet. Lead a brown bag for your teammates. Bonus points: Explain what you learned to a non-evaluator (like a parent, roommate or significant other) without putting them to sleep. Translating interesting ideas from evaluationese into lay language is a skill that improves with time.

Have additional tips and tricks to share with newbie evaluators and first-time conference attendees? Please share your ideas below.

Thanks, Ann Emery

 

Additional Resources: