Glenn: Hi, everyone. I’m Glenn Gow, Founder & Advisor of Crimson Marketing. Welcome to Moneyball for Marketing where we talk about the incredible changes happening in marketing organizations around big data and marketing technology. We feature marketing technology insights from the top marketers in the world. The reference to Moneyball is from the story of how the Oakland A’s baseball team were able to win and win and win because they figured out how to use data and technology to their advantage. If you’d like to learn about how to use big data and marketing technology and marketing to help you win visit us at CrimsonMarketing.com or email us at info@CrimsonMarketing.com. And now on to our podcast.
Today I am very pleased to welcome Cameron Deatsch. Cameron is the head of growth at Atlassian where he’s responsible for driving analytics, experimentation, demand generation, and voice of customer initiatives. So how Atlassian describe themselves? Atlassian’s mission is to unleash the potential in every team. Their products include Jeera software, confluence, Hipchat, and Bitbucket, to help teams organize, discuss and complete their work delivering superior outcomes for their organizations. Cameron it’s a real pleasure to have you here.
Cameron: Thank you.
Glenn: So you and I were talking earlier about something that’s really fascinating about Atlassian which is very successful does not have a sales organization.
Cameron: Yeah, it is true. That said, we do have a few people, but we do have people that you can call us up and if you have a common question about our pricing or licensing or how our products work, we are there to answer anything you have. So it’s not that we don’t have people that represent the business, but we don’t have a traditional inside sales, field sales, sales force based off commission. And the history of that is actually back from the beginning of the business where it really came down that the two founders fundamentally just couldn’t afford salespeople. And as they were looking to the audience, and our primary audience at that time was really technical developers which these guys were, they realized the way they made decisions about buying products was to download them, to try them, to read about them if they had questions online, and then to basically purchase things online.
Glenn: You know what’s so fascinating about that is every company I’ve talked to talks about the changing buyer’s journey, and how more and more of it is being done online, up to 70 percent. Well you guys do it all the way to 100 percent and I think that’s just fascinating, I think we can learn a lot from you, so sorry to interrupt you but let’s go ahead.
Cameron: No worries. So as our founders always said, they are never anti-sales, they are always just pro-automation. And the reality is our business depends on scale. We did not build a business to say get 5 other customers and make millions from each of them. It’s more like how to we get 50,000 customers, and across the entire organization have relatively low prices for our products. For that, we basically, although we are an enterprise software business, operate consumer like funnels. And my team is in charge of understanding those funnels all the way through our products, we run all the analytics from advertising to on our webpages to in our products, and as well as we run experiments testing experimentation at every single level from the type of ads that we run to our landing page experience to our onboarding flows, even our retention. And because of that, I actually think we’re extremely efficient when compared to a sales organization in that all of our decisions in how we improve the process of funnels, we can test and optimize daily weekly and so on, versus the time tax that comes with doing that in a sales organization, you have to train them, roll it out, have different biases. We have a much more mathematical science driven approach to how we operate our sales cycle.
Glenn: Yeah let’s talk a little bit about that. You told me you’ve got some real rockstar data scientists which is not often what you find in a sales and marketing organization. But I’d like to hear how you apply their expertise.
Cameron: The data scientists are great. I said prior that we have a few PHD level data scientists, and one had a PHD in nuclear physics, one was a pure academic, and one of my data scientist has a differential equation tattooed on his arm, I don’t know what that means for credibility, but when he starts talking, I start listening. he data scientists are great, and we built a variety of models to more or less predict future behavior of whether an evaluator will purchase, whether an existing customer will continue to renew, and so on. Advances in data science, there are very few questions we can’t really answer as long as you have enough data to feed into the models. The challenge I always have with the data scientists is to have them stop behaving like scientists which is researching and getting away into the weeds, and helping me making decisions as a business user
Glenn: Cameron you mentioned the word predictive, and many marketers that I talk to are very interested in becoming better at predictive through all sorts of things like in particular understanding their best customers, doing segmentation, optimizing offers etc. Tell us what you’re doing predictive and what you’re learning about what’s possible.
Cameron: Yeah. So we do this at almost every stage of the funnel. We’re always trying to predict the future as much as possible. One place we spend a ton of time is understanding whether an evaluator of our products is going to convert to an active user or an engaged customer, or even a paying customer. And you can evaluate, we have a variety of different products that you can evaluate in the cloud, basically subscription services from our business, or you can download and install and run them on your own servers. And when we first went down this path we actually built a predictive model based off of user behavior and the product. And what we wanted to do was run a variety of experiments all the time specifically for onboarding, and we had 30 day trials, and the reality is we would never know if an experiment was successful unless we waited a full 30 days which was not that inspiring. And it really slowed us down a lot, so the first thing we wanted to do was actually can we predict this user data much earlier, maybe at day 3 or 4, whether a customer is going to convert or not, that way we could run many more experiments throughout, and you could optimize. So we threw the data scientists that and they they took all the user behavior data, and we basically found some key trends that happen. Different activities, and we knew if we did those specific activities, that their chances that they would convert to a paying customer were well above 90 percent. So instead of waiting around 30 days, really we could just say, we basically threw their instance into our model, and popped out whether that customer would win or not. So this sped up our experimentation to 1 or 2 a month to honestly 2 or 3 per week per product.
Glenn: And that’s fascinating thing I keep hearing about digital marketing is that this concept of just AB testing isn’t necessarily the only option we have because given that you can see what’s happening in real time, you can tweak in your case a product, or you can tweak what you’re offering in real time and see how people react instead of waiting for results, you can just every time you see something work just keep doing more of that, every time something fails, stop doing that.
Cameron: It sounds simple. The way I look at it is there really isn’t from a capability whether it’s something that we built in house and there’s a variety of great tools off the shelf that you can buy now for testing, there’s really no limitation from my perspective that we can go test. I talk to marketers about like, think it up and we can basically make it happen. The challenge is how do you prioritize those, what do you even think is going to be the best, which is educated guesses at best. And then also, the more experiments that you’re running concurrently with one another or back to back with one another, the real diligence comes in the analysis on the backend where you could run an experiment for a day or two, but understanding the impact even with the predictive models could take a week or two weeks. Or what happens often is we’ll be focusing on one step of the funnel or one particular product, and see a huge success in that experiment, but it had some unforeseen consequence that we weren’t even looking at, where we changed a conversion rate for another product, or changed a different user behavior. And to understand the linkage can get really complicated really quick.
Glenn: Right, so how do you deal with that? Do you minimize the number of simultaneous experiments?
Cameron: There’s a few different approaches. First, is a process. We have a variety of different teams running experiments at different steps in the funnel. And the first thing that everyone has to do is have a standard way that we document the way we go about these experiments, and a regular cadence of communication, there should never be an excuse that we’re both running different experiments on the same customer. We have to have at least a good conversation, communication ahead of time. The second part is, as we learn these things, as we learn what more of the overlaps and challenges are, then we start to adding that to our list of checks and balance. Not just when we run the experiment, but more importantly when we do the analysis afterwards. So we know that if we see huge increase in our cloud customer conversion rates, well that actually could see a significant reduction of people that are trying our behind the firewall products. Making sure to watch both those metrics and not just one is very important. Or another piece, I have a bunch of marketing people who are AB testing the performance of a landing page by changing what variables the user has to put in or it’s their first name last name, email address. Soon we’re changing the different stages of that form. And meanwhile we’re trying to experiment on that same cohort of individuals on an onboarding flow. Well even though they’re at different steps of the funnel, they are intrinsically linked to one another, meaning that if I change someone way up high, and bring a bunch of low quality individuals into the product, that inherently going to change their behavior and how they act in the product. And we have to clearly document that and try and at least from a technology perspective, segment those individuals accordingly so we’re not doubling down on the same groups.
Glenn: That makes sense. We actually talked about attribution and the big marketing challenge of figuring out what is really working with my marketing. Talk to me a little bit about what you do there.
Cameron: I use the old joke as the standard CMO joke, I know have my money is wasted, I just don’t know which half.
Glenn: Yeah that’s a scary one.
Cameron: And my retort to that is 80 percent of your money is wasted, and we know roughly where 60 percent got wasted. Attribution in digital marketing can be a total beast. It’s one of those things where I went in when we started building this at Atlassian. I thought the simplest answer would be the correct one right, if someone clicks on an ad they’re going to convert to a customer. And since that time we decided to be much more sophisticated, and we’ve built an entire team around marketing analytics and scientists, and also data engineers with the amount of data coming in.
Glenn: What’s a data engineer?
Cameron: The data engineer is actually managing the entire pipeline of data, so all the different user events that are being triggered from clicks on ads, from visits to ur site, to visits in our products, it’s all just creating tons of data, and having some sort of structure of how that data gets pushed into our systems, we use Redshift, and more importantly setting the checks on top of that data so that it’s all flowing in as planned. Before that we’d have weeks of data disappear or not being complete or making decisions based off of that, so the engineer is really running the pipeline.
Glenn: That’s fascinating. That is not very common to find companies that have data engineers associated with the funnel, which just blows my mind. I think you also mentions you guys deal with something like 4 petabytes per day is that right.
Cameron: It’s actually terabytes, but it’s about 20 terabytes. It’s a huge amount. Maybe a 1000 blu ray discs, tons of data. Now we have to keep those organized and those data engineers have been good about that and we can pull the proper analysis out of it.
Glenn: What’s fascinating about that is most companies don’t know how to manage all that data coming in, they don’t have data engineers, and they let it go. So we have so much activity on the website, anonymous visitors coming in doing all sorts of things, and they just don’t have a way to capture that information and make sense of it. It sounds like that is a problem you’re tackling.
Cameron: Yea it was the first thing we had to do. If we wanted to answer something as simple as, did a customer who came in through a google adword search visit the site 3 times following, that eventually evaluated the product, and then bought. Understand that is a variety of different systems. Probably touching 4 or 5 different systems with different user tracking behavior on each. Internal systems, 3rd party systems and so on. Unless you have your data clean and can make the appropriate linkages between each of those steps, you’re dead in the water. You’re guessing at best, or if you’re really good you’re making correlation. We spent a bunch more money here, we saw more traffic there. Well the reality is with most digital marketing tools today, you can pretty much build the linkage one to one. And the goal is trying to get 100 percent, but if we can get 90 percent of user tracking each stage of our funnel, we’re pretty happy with our decision making ability after that.
Glenn: That’s a big number. And talk to me about advertising and attribution and what you’re doing there.
Cameron: Yeah so on the attribution piece, like I said, we started with what any marketer goes through, we started with last touch attribution which was pretty simple. But the challenge of last touch attribution especially in online marketing business is you spend more money than Google. For every dollar we spend on google advertising, we should spend another dollar on google stock and we’ll probably come out even on it. And that’s great, we know that people are searching, if they’re searching for agile project management you’re going to come to Atlassian and try Jeera. And I’ll pay for that click any day of the week because it’s going to convert very high, and it’s usually someone already looking for something we do. I always say we’re the demand generation team, not just demand capture, and ad words is really capturing demand right, someone’s looking for something you already do, you’re helping them along. So that takes me to the next step to how do we measure and quantify those higher funnel awareness tactics. Whether that’s display, radio, at home and so one. And that’s where more sophisticated attribution comes into play. And we’ve done any touch attribution which is trying to track every single click or engagement with advertisements prior to someone evaluating. We’ve done first touch and last touch. We’ve also thrown the data scientists after, so one of them read an incredible academic document on attribution, and it will take probabilistic multi touch attribution model which I won’t go into details on. The interesting piece about all of it is regardless of what attribution model you’re looking at, none of them are perfect. Everyone of them has got some sort of hole in it. And almost all the time, it will continue to say that people who are coming to your site through direct or organic means are always going to pick the best. And that from an awareness campaign tactics, tracking individual clicks and users, you can do it and invest in it, but it’s never going to be nearly as scientific as you want it to be. You want to close down to the lower touch or lower funnel tactics.
Glenn: In other words, it’s just so much harder to measure.
Cameron: Yeah. Because even with the direct 1 to 1 ratio, I know people knew the attribution, but in general when you’re getting awareness tactics, you’re really creating momentum around your brand and topics which fundamentally you’re never going to be able to measure at the individual user level. And that’s where we need to start going into larger DMAs or bigger AB tests based off geographies, and that’s kind of the net big competency that we’re trying to build up.
Glenn: Beautiful. Well I want to ask you as we wrap up a question about the future. If we look at just the next year, what do you see happening that all of us marketers should be thinking about what you guys are going to also be thinking about as it relates to the measurement of marketing.
Cameron: I know we’ll never be done. I do need to understand the balance of when do we stop, because I could have 3 marketers and 50 marketing analytic people, and then the balance is off. I want to make sure I’m getting the same ROI on my analytics and my people. As far as channels are concerned, mobile is our next big one to attack. We’re pretty happy across most online channels, but our mobile advertising, the competency is just not something we’ve built up over the last couple of years. And the reason is that with most of our products you don’t sign up from a mobile device, you don’t evaluate, you don’t purchase, nor is the primary user experience on mobile, most of the primary user experience for the majority of our products is defined for desktops at work and business. But everything you see is people are spending more time on their mobile devices, and I want to put together a comprehensive strategy so we can leverage those channels, measure them appropriately, track people across devices and so on. There’s tons of great information and tech going into that space, and I think that we just need to get well ahead of it in competency.
Glenn: Beautiful. I guess you’re lucky that the majority of people don’t buy on mobile from you today, because that’s not true of a lot of websites because most websites are already there so you have a chance to catch up and learn from the best practices there. Cameron thank you so much, this has been really helpful and I appreciate your time.
Cameron: Thank you Glenn.
Glenn: It’s been a pleasure talk to you soon.
If you like this podcast please subscribe and rate us on iTunes and tell your friends about us. You can also go to our website, CrimsonMarketing.com, and sign up for our free monthly newsletter featuring the very best of our marketing insights, featured Moneyball for Marketing podcasts, and one of our favorite features called, “Bad Marketing,” or email me at info@CrimsonMarketing.com. Thanks for listening to Moneyball for Marketing from Crimson Marketing. Have a great week and let us know if we can help you in any way.