Experimentation: What Makes or Breaks great Edtech startups 🧪 Transcend Newsletter #58

The one skill that sets founders apart in education technology.

Hi there! Alberto here, just got back to Madrid after spending two weeks in San Francisco 🌉

The Transcend Newsletter explores the intersection of the future of education and the future work, and the founders building it around the world.

We welcome 420 new readers to the newsletter since our last post. If you love reading about the future of education and work, hit the ❤️ button and share it with your friends!

Quick announcement: we’ve opened up applications for our TF9 cohort of the Transcend Fellowship! We are selecting 40 leading early-stage founders looking to find product-market fit in education & the future of work through a 5-week program, and applications close this Friday. You can learn more and apply here, and join us at our info session this Friday Oct 21 too!

Last day to apply: 🗓️Sunday, October 23

Over the past 3 years, my cofounder Michael and I have met 1000+ early-stage founders in education & the future of work, and we have noticed a pattern among founders who go on to do well, compared to those who stagnate: it’s that great founders are (almost always) great experimenters. Let me explain this.

Experiments: Not What You Think They Are

When I say experiments, don’t think of lab coats and test tubes. Think of small tests you can run as a founder, where a user’s action allows you to learn about its needs. We learned this from the one and only John Danner, and we’ve gone on to implement this idea in all of our founder programs.

John Danner @jwdanner

The reason to run an experiment every day is that experiments are your unit of learning. The more experiments you run, the more you learn about your users. The faster you run experiments, the faster you learn. The startup that learns the fastest wins their space.

Experiments help you learn about your customer much faster. And that’s the one thing founders need to excel at in the early stages: founders need to become learning machines!

This is particularly helpful for founders who haven’t yet found product-market fit. Founders often get stuck building MVPs, hiring a technical team and fundraising… but it’s often a distraction! Gagan Biyani (CEO of Maven) makes the case for skipping the MVP, and instead focusing on experiments that validate demand for your idea with a Minimum Viable Test.

Gagan Biyani 🏛 @gaganbiyani

Many founders assume the best first step in exploring an idea is building an MVP. This is wrong--there is a better way that can save you time and money. THREAD: The Minimum Viable Testing framework (a follow-up to my article in the First Round Review) 👇🏾

Experiments help your startup grow

There are many ways you can learn about your customers’ needs, but few are better than experiments. Here are the three main reason why:

#1 – Experiments help you validate before you build

The #1 pain point for founders we meet is that they are stuck building a product that nobody wants. Founders think long and hard about what tool would solve a problem they see in the world and they emerge with ✨ the solution ✨. But that rarely works – experiments let you know what is in demand, in faster and cheaper ways.

#2 – Experiments tell you what people really want

There’s a big difference between what users want versus what they say they want. If you’ve ever participated in a customer interview, you probably know what I’m talking about. The user generally wants to be nice, and sometimes they can’t even put into words what they really want.

Great experiments help you observe what people select in their actions, rather than depending on their honesty.

#3 – Experiments push you to accept you don’t know anything

When you run 5 experiments per week (this is the cadence that John recommends), you become comfortable with failure. In a good week, you probably see significant results in 1 out of 5 experiments, so you get comfortable accepting you don’t know anything (and just test every hypothesis with real customers asap).

In my experience, these three ideas are particularly helpful for founders in education, because we have a tendency to get stuck wanting to build something that we care about, but users may not. Experiments help us snap out of it, and learn what they really need.

We break down experiments into 3 types: validation, experiences, and product experiments.

1. Validation experiments 📏: these experiments help you gauge demand for a product early on (like the Minimum Viable Test). They measure interest in value propositions through landing pages, ads, social media, or posting in communities (reddit, online communities, FB Groups, private groups).

   • One great example for edtech startups is validating a learning experience (like a course) by testing the course syllabus as an MVP. You’d be suprised, but most people make decisions to participate in learning experiences because of this. Caleb Hicks, founder of Factor, shares more about this learning on this clip of our interview: he used ads with different syllabi to validate different value propositions, and doubled down on the ads that got more traction.

2. Experience experiments 🙌: these are experiments in live experiences, either online or offline (like classes, workshops, or webinars). Specific activities or concepts can be tested by the end of each experience, through engagement metrics or even assessments.

   • John initially used his experimentation framework while building Rocketship Schools, where he’d invite professors and staff to test different ideas in their daily work. We use this at Transcend all the time too – we ran weekly office hours for the first years within our community to understand what questions and challenges founders faced, and then built live sessions and curricula around them!

3. Product experiments📱: these are experiments to test specific features or communication strategies within an operational product. This applies to startups that have already found product-market fit, so they focus on increasing usage, retention or conversion.

   • Duolingo is a master at this, experimenting with dozens of landing pages for their subscription service (tweet below) and even their notification system.

Cem Kansu @cemkansu

Little known fact about Duolingo: We are obsessed with improving every screen of our products. We don't treat anything as "done". A visual example of how this obsession evolved the Super Duolingo purchase page over time:

Experiments must happen with a cadence – a rhythm with which a team operates and measures weekly experiments. We encourage our founders to get started with an Experiment Tracker (reply to this email if you want a copy of our template) that keeps all experiments in one place, and rates the results of each experiment weekly. The next week, you should double down on what worked!

Great edtech founders are great experimenters

We’ve found that experimentation is one of the best predictors of success in early-stage startups.

Founders like Neil d’Souza are masters at this craft.

Neil came up with the idea for GetSetUp (now a growth-stage learning community for senior leaners) through experimentation. He ran tiny validation experiments with companies to see if they’d be interested in hiring senior learners, and then trained the learners on the skills that resonated most with employers.

Little by little, he gained a strong understanding of his customers’ needs: senior learners wanted to feel useful, learn new skills and socialize, so he pivoted to a class-centric model, which got a lot more traction. The magic moment for GetSetUp users happened when they could learn from peers in a live class, and he found the right format by experimenting with class sizes, class length, subjects, etc.

Most days, there would be no results. But some days, these results changed the direction of the company.

How you can experiment too!

We highly encourage founders we meet to think about how they can use experiments to validate their startup ideas.

Here are four ideas to get started in your experimentation journey.

⛔ No more MVPs
If you are building an MVP right now, ask yourself “do you really need to build a product here to validate demand”? We encourage you to think of experiments (on landing pages, on communications or activities) that can help as a Minimum Viable Test, like Caleb or Neil did.

📃 Define your focus
Have you found your customer’s magic yet? Is it retention you are struggling with? We encourage founders to pin point their exact challenge right now, and focus all experiments on that one area.

🤔 Test your customer assumptions
So much of what we think our customers need, is actually in our heads. Test all your assumptions early on, so you can get a better feel for what pain point you are solving for your customer!

📝 Set up a tracker
Create a spreadsheet where you track all your experiments. You can collaborate with your team, define the hypothesis and metrics to follow, and debrief weekly! We at Transcend have an experimentation tracker of our own – if you want to access it, just reply to this email and we will send it to you!

Ok, that’s wrap!

We are big fans of experimentation. Experiments allow you to learn about your customer faster, and to validate ideas before building a product. They are quite a neat tool for founders, and we’ve found they are likely the best predictor of startup performance. We are excited to help more folks get fluent with experiments!

Get in touch with us - we want to help!

If you want some extra support getting started with experiments, you can fill out this Videoask with your questions or challenges, and we’ll get back to you this week with personalized support!

Many thanks to Neil D’Souza and Caleb Hicks for the great chats, and John for the inspiration!

The Roundup ☀️

🎓 Transcend Fellowship applications for TF9 close on Oct 23. Apply here.

🌏 Attend the Edtech Asia Summit 2022 from Oct 27-28 in Singapore, the biggest summit on the future of education and work. Buy your tickets here,

🎮 Game-based learning market setto grow to $55 billion in 8 years – Read more.

🏢 Indian edtech giant Byjus cuts 2,500 jobs. Read more.

📉 Edtech funding declined nearly 45% or $2.2B in CY22. Read more.

Network Jobs 👩‍💻

Looking for your next opportunity in the edtech + future of work space? Check out our Transcend Network Pallet to find the best job opportunities from our network:

• Full Stack Engineer @ Electives (Boston, MA)

• Learning Facilitator @ Doyobi (Remote)

• Tech Lead/Founding Engineer @ Kibo (Remote)

---

Thank you for reading!

Did you enjoy reading this piece?

Hit the ❤️ button to help us reach more awesome people like you!

Alberto