Read the document

From: To: Sent: Subject: Jerry Dischler [email protected]> Anil Sabharwal [email protected]> Fri, 3 May 2019 09:05:53 -0700 Re: Important SQV Update Cc: Prabhakar Raghavan [email protected]>, Nick Fox [email protected]>, Benedict Gomes [email protected]>, John Maletis [email protected]>, Hiroshi Lockheimer [email protected]> Thanks Anil for pushing your team and for being open to this whole line of thinking. Is there any chance we can converge on this more quickly? To elaborate: Redacted Redacted Redacted Just looking at this very tactically, and sorry to go into this level of detail, but based on where we are I'm afraid it's warranted. We are short queries and are ahead on ads launches so are short % revenue vs. plan. If we don't hit plan, our sales team doesn't get its quota for the second quarter in a row and we miss the street's expectations again, which is not what Ruth signaled to the street so we get punished pretty badly in the market. We are shaking the cushions on launches and have some candidates in May that will help, but if these break in mid-late May we only get half a quarter of impact or less, which means we need %+ excess to where we are today and can't do it alone. The Search team is working together with us to accelerate a launch out of a new mobile layout by the end of May that will be very revenue positive (exact numbers still moving), but that still won't be enough. Our best shot at making the quarter is if we get an injection of at least %, ideally ¹%, queries ASAP from Chrome. Some folks on our side are running a more detailed, Finance-blessed, what-if analysis on this and should be done with that in a couple of days, but I expect that these will be the rough numbers. Redacted The question we are all faced with is how badly do we want to hit our numbers this quarter? We need to make this choice ASAP. I care more about revenue than the average person but think we can all agree that for all of our teams trying to live in high cost areas another Redacted in stock price loss will not be great for morale, not to mention the huge impact on our sales team. I'm super proud of our pure approach at Google and don't want to poison the culture of any team, and this is why I haven't pushed harder. I also don't want the message to be "we're doing this thing because the Ads team needs revenue." That's a very negative message. But my question to all of you is - based on above - what do we think is the best decision for Google overall? In that spirit, do we think it's worth reconsidering a rollback? Or are there very scrappy tactical tweaks we can launch with holdback that we know will increase queries? (For example, can we increase vertical space between the search box/icons/feed on new tab to make search more prominent? Are there other ranking tweaks we can push out very quickly? Are there other entry points we haven't focused on that we could push on soon?) Just to be clear, the reason I haven't pushed harder on a rollback so far is because I don't want the message to be Would love your thoughts and sorry for the long email. Best, -Jerry. On Wed, May 1, 2019 at 8:20 PM Anil Sabharwal Redacted @google.com> wrote: Redacted REDACTED FOR PUBLIC FILING Ex. No. UPX0522 1:20-cv-03010-APM GOOG-DOJ-13030193

Apologies for the delay. Been traveling. Ok, I think we have a plan. Given we can't launch all of these changes at the same time anyway (otherwise we can't measure impact individually), and we don't want bad press around IO, let's roll out #1 and #2 now and get the benefits. Let's also start the search ranking experiments asap and roll those out we get the data (I'll bring it back to this group for approval first). That takes us to the week after IO anyway, at which point we are meeting and going through the menu of options as Jerry describes it. We can then review all the options and decide if we still want to roll back then. Ben - agree on all your points except maybe title before url. :) Let's discuss in the meeting in two weeks and we can agree to rollback then if need be. Does that work for everyone? Thanks! Anil. On Wed, 1 May 2019, 2:12 pm Jerry Dischler, [email protected]> wrote: Yes, agree that we should shoot for at least %. There is some indication of bias but per Atanas there is less certainty around this hypothesis than other aspects of the decline. The question in my view is what is the full set of opportunities that will get us the queries back based on our current understanding and what is the timeframe? Then from this menu we can pick the ones that add up to the target result with the lowest negative impact. Where I think the discussion gets tough is if the total doesn't get us close to the target or if the timelines are unacceptably long. On Wed, May 1, 2019 at 11:08 AM Anil Sabharwal Cheers, A. Redacted Redacted Redacted @google.com> wrote: Redacted Redacted Good news is we are aligned on the goals. What I'm struggling with is for #1, over what period of time is short term, and what cost is acceptable (user experience, risk to long term retention, team motivation). Of the Ⓡ ded, our understanding across the three teams is at least te is forecasting, and we're trying to actively understand and clawback the. We are making progress here, and I'm hopeful the search ranking improvements in Omnibox will also be a material increase in SQV, but I understand we need to do more. I just want to make sure we know that rolling back this change will be a high cost, and given the team is clearly motivated (and I have them aligned now on your #2), I would hate to make short term gains here that hurt us a lot more in the long term. I'll get you the metrics ASAP. REDACTED FOR PUBLIC FILING GOOG-DOJ-13030194

On Wed, 1 May 2019 at 22:58, Jerry Dischler [email protected]> wrote: +Prabhakar Raghavan too Thanks Anil and great news on #1-4. Can you send links to experiments so we can determine revenue impacts of the existing work? We can then compare these to the impacts for #4 and #7. Broadly we in Ads have two objectives in all of these discussions: Redacted 1. Short-term: reverse the sudden query-driven revenue loss that we saw in Q3 of last year. We're neutral to mechanism as long as we can reverse the loss which is Redacted of desktop queries. Right now it looks like we have line of sight to % queries and experiments running on the Chrome and Search side that have unknown positive impacts but I'd be willing to bet are significantly below %. It's a decent start but collectively we need to figure out how to do more and this work is urgent because we continue to face these strong headwinds in Q2. Redacted 2. Long-term: It really feels like through some deliberate efforts we can actually use entry points like Chrome to drive query growth while at the same time improving the overall user experience of the product and competitive position relative to Edge and others. We should explore this aggressively. Best, -Jerry. On Wed, May 1, 2019 at 3:16 AM Anil Sabharwal Redacted @google.com> wrote: Agree - understanding the actual revenue numbers here would be valuable. Jerry, is this something you can help with using the data we have from the 3% experiment? Logic is sound, and if this is the argument, we should reconsider #4 (ablate all) since I can't really make a strategy argument for favicons either. The issue is indeed the last point you make, but it's not just morale. It's more about giving my team a sense of ownership over the problem and an opportunity to get behind a new set of rules of engagement. I had a team meeting today and was very clear on how we need to approach these types of launches and prioritise work that was good for users and SQV. It's a cultural shift and one we absolutely need to make. But it's hard to do this by starting with an undo of work that's been live for 7 months and was approved by all teams, including ads, before the launch happened (and the impact is exactly as expected). I was willing to do it because I felt it was necessary - I wanted to help stop the bleeding and demonstrate that we are willing to be good partners and do what it takes. My team responded by doing me one better and giving us options that drive revenue and are good for users, at the same level or more than what we planned to roll back. It's going to be hard for me to say, "nice job", still roll back. The absolute $ value will always be the same so how do I show it's ever worth it, other than balancing across all feature launches Redacted REDACTED FOR PUBLIC FILING GOOG-DOJ-13030195

+Hiroshi in case this comes up in leads and he needs to defend my position (or he wants to veto it! :) A. (some will be SQV positive others will be negative, intent is to end net up)? All these little things ultimately add up to retaining Chrome users - if we lose them, we will see far greater SQV loss, and I won't have any way to get them back. On Tue, 30 Apr. 2019, 11:35 pm Nick Fox, [email protected]> wrote: Thanks for the note, Anil. And great news about 1-4! That's really nice progress, particularly #1. Couple of thoughts: Redacted Redacted, * * 1) It would be really nice to understand what the actual revenue impact is of #7. My back of the envelope is of desktop google.com revenue / year % chrome share (?) Redacted in annual revenue impact. To me that feels meaningful, regardless of what other changes claw back some revenue... 2) I agree that the user experience is better as-is, but I don't have a good sense for how much it matters to the overall Chrome experience. My sense is not that much. I bet we could spend in better ways to drive Chrome usage than through this change, for example. Redacted So, net, I think it's tricky because the absolute $ numbers are large. I think the bigger issue is the impact to your team morale. And from that POV, if forcing this down their throats causes resentment and therefore slower progress on other efforts to drive query volume, that's not good over the long term. -Nick Redacted On Tue, Apr 30, 2019 at 10:01 PM Anil Sabharwal Ben/Nick/Jerry: Thanks again for the partnership and productive conversation. Redacted @google.com> wrote: After last night's meeting, the Chrome team was able to rally and make a couple of heroic things happen. 1. With your team's help (thank you!), we were able to get launch approval to rollout two changes (entity suggest and tail suggest), that increase queries by Redacted Redacted and Redacted respectively). We are rolling both of these out live to users tomorrow. 2. We are going to immediately start experiments to improve search ranking in the omnibox (more search results and nudging search to the top). As soon as we get the data around these improvements, we will roll these out to 100% (within 2-3 weeks). REDACTED FOR PUBLIC FILING GOOG-DOJ-13030196

3. We have another SQV positive feature (query in omnibox) that we are able to roll out in the next few weeks (need to see if it's net SQV positive or if we're shifting queries from the search box on SRP to omnibox, but we're hopeful it will be net positive). 4. We will launch location entity suggest for M75 (June), which should have similar impact to the en suggest improvement. Redacted Given the above improvements, especially the % query volume increase we expect to start to see as early as this week, I would like to hold off on the rollback (#7) we discussed last night. We (Chrome) absolutely need to do our part to stop the bleeding ASAP, but given: Anil. Thanks! 1. We all agree #7 is worse for users and product usability, and the rollback runs the risk of bad PR/user sentiment during the week of 10; To be clear, knowing what we know now, we likely would have not made the decision to launch these omnibox changes in the first place until we were able to balance them with other launches, and so I think we're in a good place with my team and their understanding for how we want to move forward. But given this has been live for 7 months and is very usable visible (see this reddit thread where users in our ablation experiment noticed and called it a bug!), and we have a different way to help (partially) stop the bleeding right now, I no longer feel the cost outweighs the short term benefit. Redacted 2. The latest ablation experiment data for #7 only showed a query volume increase of Redacted) ⁹%; 3. The improvements above should more than cover this and will start rolling out immediately, I believe this to be the right trade-off. I acknowledge another viewpoint is we should do all of the above to maximize SQV, including the rollback, but in the long run I think this will actually hurt the inertia and motivation the team has to build features that are good for users AND increase SQV. By balancing these launches, we send the right message to the team on how we want to operate and partner. If you disagree, please let me know and we can jump on a call ASAP to discuss. It's important to me we work together to make these calls. P.S. All the experiments for the NTP around fakebox and shortcut ablation are still running and we will have the data in the next week. REDACTED FOR PUBLIC FILING GOOG-DOJ-13030197