Ex.
No.
UPX0522
1:20-cv-03010-APM
Redacted
REDACTED
FOR
PUBLIC
FILING
GOOG-DOJ-13030193
From: Jerry DischlerRedacted@google.com>
To: Anil Sabharwal
Redacted
@google.com>
Sent:
Fri,
3 May 2019 09:05:53
-0
700
Subject: Re: Important
SQV
Update
Cc: Prabhakar Raghavan
Redacted@goog
Garnes
John [email protected]>, Hiroshi Lockheirner
Thanks Anil for pushing your team and
for
being open to this whole line
of
thinking.
Is
there any
chance we can converge on this
more
quickly? To elaborate:
Just looking
at
this very tactically, and
sorry
to
go
into this level
of
detail, but based on where we
are
Redacted
I'm
afraid it's warranted. We are short
%
queries and are ahead
on
ads launches so are short
Redacted
% revenue
vs.
plan.
If
we don't hit plan, our
sa
les
te
am doesn't get its quota for the second
quarter in a row and
we
miss
the
street's
expectations again, which is not what Ruth signaled to
the street
so
we get punished pretty badly
in
the market.
We
are shaking the cushions
on
launches
and
have some candidates
in
May that will help, but
if
these break in mid-late May we
only get half a quarter
of
impact
or
less, which means
we
need Redacted %+ excess to where
we
are
today and can't do it alone. The Search
team
is working together with us
to
accelerate a launch
out
of
a
new
mobile layout
by
the end
of
May
that will be
very
revenue positive (exact numbers
still moving), but that still won't be enough. Our best shot at making the quarter
is
if
we
get
an
injection
of
at least ideally
Redacted%
, queries ASAP from Chrome. Some folks on our side
are
running a more detailed, Finance-blessed, what-if analysis on this and should be done with that
in a couple
of
days, but I expect that these will be the rough numbers.
The question we
are
all faced with is how badly
do
we
want to hit
our
numbers this quarter?
We
need to make this choice
ASAP
. I care more about revenue than the average person but think
we
can
all
agree that
for
all
of
our
teams
trying
to
live in high cost areas another Redacted in stock
price loss will not be great for morale,
not
to
mention the huge impact on our
sa
les team.
I'm super proud
of
our pure approach
at
Google and don't want to poison the culture
of
any team,
and
thi
s
is
why I haven't pushed harder. I also don't want the message to
be
''we're doing this
thing because the Ads team needs revenue." That's a very negative message.
But
my
question
to
all
of
you
is
- based on above - what
do
we think
is
the best decision for Google overall?
ln
that spirit, do
we
think it's worth reconsidering a rollback?
Or
are there very scrappy tactical
tweaks
we
can launch
with
holdback that we know will increase queries? (For example, can we
increase vertical space between the search box/icons/feed
on
new
tab to make search
more
prominent? Are there other ranking tweaks
we
can push out very quickly? Are there other entry
points
we
haven't focused on that we could push on soon?) Just to be clear, the reason I haven't
pushed harder on a rollback
so
far
is
because I don't
want
the message to
be
Would love your thoughts and sorry for t
he
long email.
Best,
-Jerr
y.
On
Wed
, May 1, 2019 at 8:20 PM
Anil
Sabharwal
wrote:
Apologies for the delay. Been traveling.
Ok, I think we have a plan. Given
we
can't launch all
of
these changes at the
same
time
anyway (otherwise
we
can't measure impact individually), and we don't want bad press
around
IO,
let's roll out # 1 and
#2
now and get the benefits. Let's also start
the
search
ranking
experiments asap and roll t
ho
se
out once we get the data (I'll bring it back to this group for
approval first). That takes us to the week after IO anyway, at which point we are meeting and
going through the menu
of
options
as
Jerry describes it. We can then review all the options
and decide
if
we still want
to
roll back then.
Ben - agree on all your points except maybe title before url. : ) Let's discuss in the meeting in
two weeks and we can agree to rollback then
if
need be.
Does that work for everyone?
Thanks!
Anil.
On Wed, 1 May 2019,
2:
12
pm
Jerry Dischler, [email protected]> wrote:
Yes, agree that we should shoot for at least
Redacted
%.
There
is
some indication of bias but per
Atanas there
is
less certainty around this hypothesis than other aspects
of
the decline.
The question in
my
view
is
what is the full set
of
opportunities
that
will
get us the queries back
based
on
our current understanding and
what
is
the timeframe? Then from this menu we can
pick the on
es
that add up to the target result with the lowest negative impact. Where I think
lhe discussion gels tough
is
if
the
total doesn't gel
us
close
to
the target or
if
the limelines are
unacceptably long.
On Wed, May 1, 2019
at
11
:08
AM Anil Sabharw·al [email protected]> wrote:
Good news is we are aligned on the goals. What
I'm struggling with
is
for #
1,
over what
period
of
time
is
short term, and what cost
is
acceptable (user experience, risk to long term
retention,
tea1n
motivation).
Of
the
Redacted%,
our understanding across the three teams
is
at
least
Redacted%
is
forecasting, and we're trying to actively understand and clawback the % We
arc making progress here,
and
I'm hopeful the search ranking improvements
in
Omnibox
will also be a
n1aterial
increase
in
SQV, but I understand we need
to
do more. I just want to
make
sure
we
know that rolling back this change will be a high cost, and given the team is
clearly motivated (and I have them aligned now
on
your #2), I would hate to make short
term gains here that
hurt
us
a lot more in the long term.
I'll
get you the metrics ASAP.
Cheers,
A.
Redacted
REDACTED
FOR
PUBLIC
FILING
GOOG-DOJ-13030194
On Wed, l May 2019
at
22:58, Jerry Dischler [email protected]> wrote:
+ Prabhakar
Raghavan
too
Thanks Anil and great news on #1-4. Can you send links to experiments so we can determine
revenue in1pacts
of
the existing
work?
We
can then compare these
to
the impacts for #4
and #7.
Broadly we
in
Ads have two objectives
in
all
of
these discussions:
1.
Short-te1m: reverse the sudden query-driven revenue loss that
We
saw
in
Q3
of
last year.
We're neutral to mechanism as long
as
we can reverse the loss which is Redacted%
of
desktop
queries. Right now
it looks like we have line
of
sight
to Redacted% queries and experiments
running on the Chrome and Search side that have
unknown
positive impacts but I'd
be
willing to bet are significantly below Redacted%. It's a decent start but collectively we need to
figure out how to do more and this work is urgent because we continue to face these strong
headwinds in Q2.
2.
Long-term: It really feels like through some deliberate efforts we can actually use entry
points like Chrome to drive query growth while at the same time improving the overall
user experience
of
the product and competitive position relative to Edge and others.
We
should explore this aggressively.
Best,
-Jerry.
On Wed, May
1,
2019 at
3:
16
AM Anil Sabharwal
wrote:
Agree - understanding the actual revenue numbers here would be valuable. Jerry,
is
this
something you can help with
using
the data we have
fro1n
the 3% experiment?
Logic
is
sound,
and
if
this
is
the argument, we should reconsider #4 ( ablate all) since 1
can't really
make a strategy argument for favicons either.
The issue is indeed the last point you make, but it's not just morale. It's more about giving
my
team a sense
of
ownership over the problem and an opportunity
to
get behind a new set
of
rules
of
engagement. I had a team
meeting
today and was very clear on how we need to
approach the
se
types
of
launches and prioritise work that was good for users and SQV.
It's a cultural shift and one we absolutely need to
make.
But it's hard
to
do this
by
starting
with
an
undo
of
work that's been live for 7 months and was approved by
a11
teams,
including ads, before the launch happened (and the impact is exactly
as
expected). I was
willing to
do
it because I felt it was necessary - I wanted
to
help stop
the
bleeding
and
demonstrate that we are willing
to
be good partners and do what
it
takes.
My
team
responded by doing me one better and giving us options that drive revenue and are good
for users,
at
the same level or more than
what
we
planned to roll back.
It's
going
to
be
hard for me to say, "nice job", still roll back. The absolute
$ value will always be the
same
so
how
do I show it's ever worth
it
, other than balancing across
all
feature launches
Redacted
REDACTED
FOR
PUBLIC
FILING
GOOG-DOJ-13030195
(some will be SQV positive others will be negative, intent
is
to end net up)?
All
these
little things ultimately add up to retaining Chrome users -
if
we
lose them, we will see far
greater SQV loss, and I won't have any
way
to get them back.
+ Hiroshi in case this comes up in leads
and
he needs to defend
my
position ( or
he
wants to veto
it!
: )
A.
On
Tue,
30
Apr. 2019,
11
:35
pm
Nick
Fox,[email protected]>
wrote:
Thanks for the note, Anil. And great news about 1-4! That's really nice progress,
particularly #
1.
Couple
of
thoughts:
1)
It
would be really nice to understand
what
the actual revenue impact is
of
#7.
My
back
of
the
envelope is Redacted
of
desktop google.com revenue /
year
*
Redacted%
chrome
share
(?) *
Redacted in annual revenue impact. To
me
that feels meaningful, regardless
of
what other changes claw back some revenue ...
2) I agree that the user experience is better as-is, but I don't have a good sense for
how
much it
matters to the overall Chrome experience.
My
sense
is
not that much. I
bet
we could
spend
Redacted in better ways to drive Chrome usage than through this change, for
example.
So, net, I think it's tricky because the absolute $ numbers are large. I
think
the bigger issue
is
the impact
to
your team morale. And from that POV, if forcing this
down
their
throats
causes resentment and therefore slower progress on other efforts to drive query volume,
that's not good over the long
term.
-Nick
On
Tue, Apr 30, 2019 at 10:01
PM
Anil Sabharwal [email protected]> wrote:
Ben
/Nick/Jerry:
Thanks again for the partnership and productive conversation.
After last night's meeting, the Chrome team was able to rally and make a couple
of
heroic
things happen.
Redacted
1. With your team's help (thank you!),
we
were able to get launch approval to
rol1-
out two changes ( entity s·uggest and tail suggest), that increase queries by
Redacted¾
Redacted% and Redacted% respectively).
we
are rolling both
of
these out live to users
tomorrow.
2.
We
are going to immediately start experiments to improve search ranking
in
the omnibox (more search results and nudging search to the top). As soon
as
we get the data around these improvements,
we
will roll these out to 100%
(within 2-3 weeks).
REDACTED
FOR
PUBLIC
FILING
GOOG-DOJ-13030196
3. We have another SQV positive feature (query in omnibox) that we are able
to roll out in the next few weeks (need to see
if
it's net SQV positive or
if
we're
shifting queries from
the
search box on SRP to omnibox,
but
we're
hopeful it will
be net positive).
4.
We will launch location entity suggest for M75 (June), which should have
similar impact to tbe entity suggest improvement.
Given the above i1nprovements, especially the
Redacted%
query volun1e increase we expect
to start to
sec
as early as this week, I would like to hold
off
on the rollback (#7)
we
discuss
ed
last night. We (Chrome) absolutely need to do our part to stop the bleeding
ASAP, but given:
1.
We
all agree #7 is worse
for
users
and
product usability, and
the
rollback
run
s
the risk
of
bad PR/user sentiment during the week
of
IO;
2.
The
latest ablation experiment data for #7 only showed a query volume
increase of
Redacted%;
3. The improvements above should more than cover
th
is and will start rolling out
imm edi ate 1 y,
I believe this to
be
the rig
ht
trade-off. I acknowledge another viewpoint is
we
should do
all
of
the above to maximize SQV, including
the
rollback, but in the long run I think
this will actually hurt the inertia and motivation the team
has
to build features that are
good for
users
AND increase SQV.
By
balancing these launches,
we
send
the right
message to the team on
how
we
want to operate and partner.
To be clear, knowing what
we
know now,
we
likely would have not made the decision to
launch these omnibox changes in the first place until we were able to balance
them
with
other launches, and so I think we're in a good place with
my
te
am
and their
underslanding for how
we
want to
move
forward.
But
given this has been live for 7
months and is very usable visible (see this reddit thread where users in our ablation
experiment noticed and called
it a bug!), and we have a different
way
to
help (partially)
stop the bleeding right now, I
no
longer feel the cost outweighs the short term ben
ef
it.
If
you disagree, please let 1ne
know
and
we
can
jump
on
a call
ASAP
to discuss. It's important
to
me
we work together to make these calls.
Thanks!
Anil.
P.S.
All
the experiments for the
NTP
around f
akebox
and
shor
t
cut
ablation are still running and
we will have the data
in
the next week.
Redacted
REDACTED
FOR
PUBLIC
FILING
GOOG-DOJ-13030197
