The Importance of Graceful Rule Breaking When it Comes to Research

Dr. Kirsten Lee Hill is a  researcher, creative, and entrepreneur. She works with people and organizations who have an idea for how to make society better and want to use research to amplify their impact. She supports them by making research accessible while also centering their personal wellbeing so they can secure more money, earn credibility, and confidently advocate for their vision to change the status quo.

Since starting her company in 2016, Kirsten has coached over 150 entrepreneurs on measurement and evaluation;  helped raise millions in grant funding for research and innovative ideas; and partnered with global leaders in innovation such as Virgin Unite and the Bill & Melinda Gates Foundation.

Kirsten earned her Ph.D. from the University of Pennsylvania where she studied school turnarounds and led the development of citywide surveys to measure alternative indicators of success in schools. Having worked on everything from randomized control trials to designing alternative teacher education programs, Kirsten is an expert in creatively leveraging traditional research expertise to support community-driven change. She currently serves as Co-Director and Co-PI of a Bill & Melinda Gates-funded national study of professional learning in addition to running her boutique consulting firm which is committed to making research a tool for good in the world while centering the role of self-care. Through her podcast, Graceful Rulebreakers, Kirsten inspires her listeners to seek sustainable change, using her own self-care journey as a blueprint to help others.

Below is an interview I recently had the honor of hosting to hear the many creative insights Kirsten has about research and it’s importance in the world of business.

 MELISSA:   I find that many thought leaders who are writing books don’t pay attention to all the potential that research has for them. In other words, there aren’t enough books that are well-researched.

KRISTEN:   Oh, absolutely! A lot of people think research is too much work or could be a lot to undertake or it isn’t relevant to what they do. Or maybe they feel like they’ve done their research and don’t need to do more.

But to me, research is such a powerful tool for amplifying and building a case for the story you’re telling. The more you’re able to look at what is going on in your space in a strategic way, the deeper knowledge you will have and be able to share. Conducting research will build your confidence as a thought leader. It will also build your credibility because you’re not just speaking in the vacuum of your experiences. You’re also able to point to scientific studies and other things that other people have experienced. That’s really important.

I mentioned a strategic way because there’s a tendency to do almost careless research in the sense that, if you don’t really sit down and think, “What is the question I want to answer?” And then ask: How could I best get the answer to this question? Often authors go bouncing around on the internet as their research approach—googling, pulling up different websites, and learning different things, but they’re not necessarily credible or representative of truth, or of multiple opinions converging on a truth. So it can be easy to get lost.  or to read one article that seems really compelling and fascinating, and you think, Oh, this must be true.

For me, I always talk about research as something you should be right-sizing to your organization to make it more accessible to people.

MELISSA:   I like that, right-sizing. Expand on that a little bit, because I work a lot with authors in Fortune 500 companies, but then these other fast-growth smaller companies…Talk more about research in various sized organizations.

KIRSTEN:   I was classically trained to research. I got a Ph.D. and ran big multimillion-dollar, multi-year studies. That’s big research with big dollars spent. Here, you have lots of time and resources. You also need university partners. But that’s not true at all, which is why I really got on this idea of right-sizing because if you’re a smaller organization, you could do smaller research.

For example, if you’re a startup and you’re 1, 2, or 5 years into running your idea—your program, you don’t necessarily need to spend a couple of million dollars on research. You could do a much smaller-scale project that still gets you useful information. I’ve done a lot of work with an organization called 4.0. They’re in the education space, and they invest very small amounts of money—between $600 and $5,000 in ideas that people have to change education. That’s the budget. You’re getting maybe a couple of thousand dollars, and your job is to test your idea to see if it works. It was so fun working with this organization—getting to educate and partner with people around the question of, “What does success for me look like on this really small scale? What do I need to see happen in the next week or months?” Because these are really short studies…to make me think, “This is a good idea. We should keep going.”

And what was amazing was that everyone could do it. We had 45 or 46 different teams that came up with plans, ideas, surveys, and Observation Rubrics, and assessments of whether or not they work. And they did it! They did it on a small budget and on a short timeframe, and they had access to researchers to partner with. But literally, they did it by themselves.

Right-sizing research is that instead of saying, “Well, I don’t have the money” or “I should wait until my organization is bigger or more thought-out,” you can actually think about, “What would be the right size for me now? What are the questions I have now, and what information could I get to answer them quickly?”

Especially in business and society, who wants to wait three to five years for a huge study to tell you what to think? If you can get some good information now, let’s get it, and make better decisions.

MELISSA:   And in a nutshell, that is the power of good and effective research, right?

KIRSTEN:   Yes. And I love to explain research. So “Kirsten’s definition” is asking a question and then making an intentional plan to answer it. And that’s it. Research is about questions, and the planning piece is key because you do want to be strategic about the information you get.

I’ll use this example: If you’re working with people who are millennials and you want to collect research about what millennials like, and then you send out a survey and the only people who answer are people from Gen Z, that was a bad plan. That doesn’t get you the information you need. So really having that intentional planning piece so you get your questions, and get your plan really is that simple. So if people spent more time getting clear on what they want to know and thinking through, “What’s the best way I could answer this question? Who should I talk to? What data should I be looking at?” I think people would have such rich, powerful data they could use to tell a story or make an argument, which are things that thought leaders want to be able to do effectively.

MELISSA:   I often find that authors don’t realize the usefulness, the importance, maybe even the necessity of conducting research. Perhaps one of the reasons is that they don’t realize what’s involved; they don’t think about it enough.

That’s why I really appreciated hearing about you and seeing the now hearing about the approach that you’re taking. Your process is commendable and incredibly important, and also an important element to include when writing a book.

KIRSTEN:   Absolutely. It’s like if one person tells you something, is it true? Versus if you ask lots of different people and you see statistics that support it, then it builds credibility. And in the world we live in today, especially for thought leaders who are mindful of this, there is so much misinformation out there. To me, thought leaders have a responsibility to be ethical and transparent if they’re saying things they believe are true.

I go through this exercise with people where I say, “Whenever you see a fact, or what some people say is true, there are five things you have to ask. The first question should be, who did they get this information from? You see surveys that say, “80% of the people will think this is a good idea.” I want to know who you talked to. Were they people like me, people in a different country? People younger, older? Did you only talk to males? There are all of these contextual things that go into whether something is actually a credible piece of evidence.

Because as thought leaders we get to say big, exciting things…inspire people. But then I think it’s on us to dig in and say, “here’s how we know this is true. Here is some good evidence. We talked to a bunch of different people.” Or if you didn’t talk to a bunch of different people, just say, “I’m going to level with you. I only talked to teenagers who live in rural America, and this is what I found.”

Then I say, “Well, you’re more credible because at least you didn’t lie and try to pretend that applies to everyone in the world.”

MELISSA: So what are three pitfalls that can make your questions too complicated?

KIRSTEN: I think the most common one, and this is something I see whether the question is verbally being asked, or in a survey or form – is covering way too many things at once. I have a rule: one question, one topic. I don’t want to see the word “and” in any question, because now you’re asking the person to start thinking about multiple things at once, and that muddies their answer. They might forget to answer half of it.

So in-person or on the phone in an interview, if you ask a really long question, “Tell me about your experience growing up, and also what inspired you to start this business? And also how are things going now?“ You just asked me three separate things all at once. That’s harder for me to process and give you a thoughtful response and make sure I’m speaking to everything you want me to speak to, versus if you just ask me one thing at a time.

The benefit of being in live conversation with someone is if someone forgets something, you can say, “Oh, you didn’t actually mention anything about your childhood. Could you speak to that?” But if you’re collecting information on a survey, you don’t have that luxury. However, you wrote it is how they’re going to read it, and you don’t know how they’re going to interpret it. This is such a bite-sized basic example, but someone might ask, “Do you like reading and writing?” and give me a check-box to say Yes or No. And that’s actually a very confusing question. If I love writing and I hate reading, do I pick Yes or do I pick No?

Because you’re asking me to evaluate two things, and only giving me one space to do that. So I think really simplifying our questions so that we are asking one passage at a time allows people to expound and just focus on that topic – it just would be super-helpful for everyone, and it’s something not that many people do. So that is always top of mind as something I think we could do better.

Along similar lines, we should be very aware of the language that we use. When we do surveys for research studies, we have a rule where if a third-grader couldn’t read the question and understand all the words and what we’re asking, then it’s too hard. Even if you’re surveying adults with doctorate degrees, you still want it to be third-grade reading level. And the last thing about it is you just want to be simple. If you have to use a word that maybe people wouldn’t know exactly what you mean by it, include a definition. That’s super-important.

Another example similar to that is, there are a lot of people who ask if someone is satisfied with something. I hate seeing that question and I hate being asked that question, because what does it actually mean to be satisfied?

How I think about satisfaction is not necessarily the same as how you think about it, so it’s a throwaway question. If you want to know something, I encourage people to get really specific about what they actually want to know.

If I ask someone if they’re satisfied, what I really want to know is if they thought the feedback I gave them was useful. So why not just ask that instead? I think we get caught up in the word “satisfaction.” There are also a lot of buzzy words and vague language that are very common to use. But that’s confusing, and then you don’t really know what people are thinking of when they respond. Breaking things down can be super helpful in getting better information.

And lastly, I would say is to always test your questions out before you take them to the field and ask a bunch of people. I know from personal experience nothing is more humbling than sitting down with someone and taking them through their questions and hearing them narrate their internal dialogue about how nothing I’m asking makes sense. And it’s so helpful! (CHUCKLES)

MELISSA:   Yes, it is. I love the term graceful, and your idea of being a graceful rule-breaker. Now I’m thinking with your model, am I asking her too many questions here? (LAUGHTER) I will ask specifically, tell us about graceful rule-breakers and rule-breaking?

KIRSTEN: I was giving a talk to Tulane University, where I went for undergrad. I said, “I guess I’m a graceful rule-breaker,” and I thought, Ooh, that’s a fun terminology I just invented on the spot in this talk. To me, “graceful rule-breaking,” especially for thought leaders and others who want to change society; we’re often working in systems that we think are broken and don’t necessarily work that well, and we want to make them different.

In my experience, I haven’t found it particularly effective to say to those with whom I work, “Hey, this sucks. You’re doing everything wrong; we should just start from scratch.” That has never been well-received in any room I’ve been in. Whereas if my approach is more collaborative, more graceful, and I’m able to suggest small tweaks and build rapport and connect with people—to take what’s valuable to them and then tie that into bigger changes we might make for the system, that has been so much more effective for me.

MELISSA:   That’s a very good idea. Maybe people get over-zealous. They want to do more. But I’ve never heard it presented exactly the way you presented it. I think that’s something to think a lot about, then the word graceful fits right in here.

KIRSTEN:   And I love it; it’s just this idea that you can have a softer and more graceful, elegant approach, but still be working to fix things that are messed up. I think I’m saying, “Yes, this is messed up, but the way to change it is to actually engage with it,” and I really learned that when I was in graduate school. I started off with my graduate school career attending a ton of protests, which is not really that graceful.

I had a lot of angry signs about education reform when I was in Philly. I hated how they were running Philly schools, and was very vocal about it. And I never got invited to any meetings to share my opinion. No one saw my sign or said to me, “Hey! Great idea! We should definitely get rid of standardized tests.”

Then through my graduate studies, I got invited to manage a partnership between the University of Pennsylvania and the school district of Philadelphia. I remember going into this meeting, and it was the first time in years that I’d ever been inside the school district. Because I was always outside of it, yelling with a sign. And I thought, “This is so different. I get to go inside.”

By getting inside the meeting with the leaders, I remember thinking, “Wow. These are not bad people. I’ve spent a couple of years yelling at them outside their office windows about how people were making stupid decisions, and they’re actually not bad people.”

Having the opportunity to speak with them and then to run a research study with them, and really collaborate on the issue of, “How do we make our schools better?” That to me was this graceful rule-breaking moment, where I could protest outside and be completely shut out of the process, or I could have some humility, go into the building, and respectfully collaborate with people. And we got a lot of incredible things done. This new experience opened my eyes to a different way of approaching change.

It’s not that there’s never a time and space for protests and freedom of speech. Those are great things. But for me, those weren’t putting me in a position to make change happen, whereas reaching across the aisle, going in with a partnering mentality, and then being in those meetings and being able to push back on things that seemed unfair, that was actually productive.

MELISSA:   Very well put on your part, and to me, gives any change-maker or wannabe changemaker a true north. Because there are so many issues with school reform, have you taken it to another level? You obviously work on a lot of different projects, and you talk to a lot of people. Any next steps in the school reform area?

KIRSTEN:   Yes, it’s so interesting. That was more cutting-edge research in the sense that we were partnering with the school district. But we still have our major research findings from that study, which ended in 2016. It unfortunately still hasn’t been published because the peer review process takes so long. But any chance I get, I love talking about it. I have a book chapter on the work we did there. But only about ten people have probably read that chapter as it’s in an academic book that costs way too much money.

There’s this whole research accessibility issue that really lights me up, because it’s just like, “Don’t you even read the stuff we produce?” But what I took from that experience is, we investigated a lot into making the idea of successive schools more complex. Typically schools are successful in policy-makers’ eyes if they’re doing well on standardized tests. And for that study, what we did was designed surveys that teachers and families and students and school leaders took to report on different aspects of their school. What does it look like in terms of relationships and safety? And what type of instructional strategies do you use, and how do you engage with families?

We tried to paint this more holistic picture of what a good school looks like, and layer on top of that the how-to. How do you create change? I’m a total nerd. I’m really into something called policy attributes theory, and it was developed in the ‘80s. It’s a theory about how you effectively create change, and it says if you want an idea to work, you have to be really specific, you have to align it with what’s going on in society, the organizational goals; you have to get a lot of science, you have to have enough money. You have to have stability over time, and you have to either provide rewards or penalties for people who don’t buy-in.

So we analyzed all these reform efforts using those indicators, and it’s just been a really interesting framework to unpack change. I’m actually running a study now – I’m co-director of a study of professional learning for The Gates Foundation, and we’re using the same ideas. So the things we came up with back in 2014 in that study are still useful frameworks for looking at change in schools now. It’s been interesting.

Melissa: What’s been the most exciting, though?

Kirsten: I mentioned 4.0 and doing work with other innovative education. There is this grassroots movement to create change in schools. And I think that’s what’s really cool about research and making research more accessible is that we’re no longer in a space where a university has to say, “This is a great, effective way to run a school.” I think now teachers and families can design a way to run a school, and they can collect information that shows why it’s good, and I think that’s really exciting and empowering. And I have loved the opportunity. I coached 150 entrepreneurs over the past three or four years, specifically in education, on how they’re bringing change and how they’re measuring success. And I was able to do that because of the lessons I’ve learned in graduate schools and my studies, and to me, really the most exciting part was to be able to leverage my more traditional knowledge and allows it to be used in a much more radical way.

MELISSA:   I love it. I love that you’re taking thought leadership and making it more practical. There are so many people who put that development of thought leadership on a shelf. And you didn’t.

I know you also have a podcast that you do. Is that also from you getting the word out? You are, for me, the most interesting thing here is the way you communicate these ideas. It almost feels like an invitation to start practicing this methodology, way of thinking differently and not discounting yourself…you’re making it very accessible and inviting for people to look into these things.

I love what you’re doing around the school reform. I also know you are still consulting and have a podcast I’m sure readers here would love to discover. You definitely have shown us the wonderful philanthropic elements of your work.

KIRSTEN:   That’s my goal. I like to think about the theme in my business because I do have my hands on a lot of different things. It really is this idea that people matter and you as an individual, you matter. So when I’m doing research, it’s because I want more voices to be heard because they matter. And when I’m doing my podcast, I talk about research sometimes, because that’s a huge part of my life. But it’s very much around the idea of how do you cultivate this inner knowing that you matter, and how do you make valuable contributions to this world? And how do you hold onto that if you are a changemaker?

People will very often disagree with you or think what you’re doing is too much, or doesn’t make sense. Or is too different than the way things are. So to me, there’s a personal journey of how do I sustain in this work and cultivate my values, and trust myself to continue being authentic and show up?

I started making my podcasts really just for myself. I would record it in the voice memo app on my phone, and I had all these episodes of personal reflection on my life, trying to do my part to make the world a better place. I launched in July of last year, and I had this realization that it’s really kind of selfish if you have a story that could benefit people, which I honestly believe everyone does – it’s kind of selfish to not share it because you’re scared of how people might react.

So I thought, “You know what? I’m putting the podcast out!” I really just went all-in on it and found it to be very therapeutic for me. I love creative endeavors. It’s fun for me. I like being able to share my story, and it allows me to practice what I preach. I go around all day telling people, “You’re so important. You should share more about what you do. You never know when you’re helping or inspiring someone.”

MELISSA: I like to call it what you are doing, “The Great Invitation.” When someone like you who put a lot of time, effort, thought leadership, and especially heart into their work then goes out and does all kinds of wonderful things, and invites others to do something similar with their own unique talents for the world and for themselves, of course, that is very powerful. I would say those who do that make it all about us rather than all about them, which it should be. It is all about us being together, working together, caring about each other together. So you’re one of those wonderful souls, it’s great that you go out and invite others to do similar things. Whereas there are those who haven’t gotten that memo yet!

KIRSTEN:   Yes. I think there are also people who have a scarcity mindset. “If other people are successful, I’m no longer successful.” It makes me sad because nothing makes me happier than seeing other people feel confident and empowered and excited. That’s great! We can all feel that way. Plenty to go around! So what’s coming up for you in your work?

KIRSTEN:   Today is a really exciting day because just a couple of hours ago I relaunched my website, which is very exciting for me.

MELISSA:   And what’s the URL for the website?

KIRSTEN: It’s KirstenLeehill.com And to me, this represents the next big thing for me; I worked really hard on the branding and just making my message even more accessible. I like to make a Mean Girls quote because I love pop culture. “I’m not a regular researcher. I’m a cool researcher.” I want people to feel invited in and that they can access me and access this knowledge. So I’ve really pivoted to trying to produce much more creative content. I’m 33. I got on TikTok. I’ve been making one-minute TikTok videos on Research This, which has been an interesting experience.

I have a YouTube channel with 5 or 10 minutes of, “Here’s how to ask better questions. Here’s what you could be doing in research.”

MELISSA:   I love the website. I just looked at it. I’m really resonating with your offer to provide people with one-to-one mentoring, not coaching. You call it a complimentary coaching session, but it’s your mentorship program, and I really like that. That’s very special. I wish you the best, Kirsten, and look forward to seeing your exceptional research skills continue to help thought leaders. I know I’ll be referring you to my authors.

***

An additional article you might find useful: