Finding Drop In Organic Search Traffic- Case Study Of Client
This case study is addressing several concerns from stakeholders and the concerns were as follows:
- Drop in organic traffic without any changes to the website
- Majority of the drop is mainly for one specific region
- A significant drop in conversion rates
- The drop is across all channels and comparatively higher drop is from organic channel
Organic Traffic can narrow down for a myriad of reasons, sometimes it could be for some major reasons and predominantly due to the implementation of errors and sometimes due to some external factors, which again might impact to a greater proportion or nothing at all.
Below is such an instance where one actually isn’t oblivion about any particular reason of the sudden drop in Organic Traffic, this becomes an even greater problem especially since the organic channel is the biggest source of traffic and the revenue is completely dependent on the Organic Traffic.
I am presently dealing with an account which has a widespread global presence whose revenues are stratospheric and any minor problem can affect largely on the overall revenues for the month, it’s a trying time for the sales and marketing professionals when there are a few predictable cases and many unpredictable ones those which are impacting the organic performance of a website.
Let’s get to the root on what is currently happening on the account:
- Duration: August 1-20th 2018
- Drop in Traffic- 10%
- Drop in conversion rate by 17.5%
- Transactions by 27%
- Revenue by 16%
I reckon the following reasons for the depletion:
- Since the major drop is from India, it could be a seasonal effect.
- Drop in Traffic due to the fall in the average position of keywords.
- Drop in traffic that comes from the brand terms, as paid marketers are bidding for it and take some of it.
- Any technical glitches/blunders either by the marketing teams or development or both on the systems.
Anything else?
The product availability for the considered duration:
If the product availability is displayed as out of stock, of course, there will be a drop in CTR and in turn drop in the traffic to the website. The product availability snippet can be replaced with the schedule snippets for training, the entertainment industry, rating and reviews for other industries, and something else for some other industry, and its absence will impact the CTR (Click through rate) and in turn the traffic. Anyways, let’s not get into it! Let’s leave the matter to the respective product owners or inventory managers or operation teams to dig into it.
Note: With the drop in CTR search engines push your website down and later it would take some time to retrieve it to its actual positions despite the availability of the product.
Talking about the 4 reasons for the major drop in Traffic:
1. Since the major drop is from India, it could be a seasonal effect:
August is the month of Independence for India, and along with that the land of diverse festivals is also celebrating Bakri- Eid, both the occasions fall in the mid-week and ergo there is a high possibility of techies/professionals planning their weekends and festivities, that could have been the reason for the drop in searches on internet or on search engines like Google, Bing and more.
Let’s dig a bit deeper and understand if our presumption is true:
Week 2 Vs. Week 1
Some sample data from the webmaster tools
Some noticeable dip in the horizontal lines exactly the day of Independence day, and it’s not a sharp dip, it’s actually linear over that week. Let’s go a bit deeper to that date, I also observed the Avg position of keywords has come down by 3% and this would have been the actual reason for the drop.
Week |
Impressions |
Clicks/Traffic |
CTR |
Avg Pos |
Week1 Aug |
319539 |
9463 |
2.96% |
24 |
Week2 Aug |
283033 |
8541 |
3.02% |
24.6 |
Diff |
-36506 |
-922 |
0.0006 |
0.6 |
%Change |
–11% |
-10% |
2% |
3% |
2. Drop in Traffic due to the change in keywords SERP and CTR:
Case1: No Change in Avg position of keywords but a gradual drop in impressions:
This clearly implies that the number of searches has come down from the region which is prominently contributing for impressions, here, in this case, India is the major contributor and hence there is a seasonal impact.
For Data Privacy reasons the numbers are alerted in the table and across the article, the altered number has a proportionate difference across the complete set and dimensions,
Queries |
Week2 Clicks |
Week1 Clicks |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
Week2 Avg Pos |
Week1 Avg Pos |
Avp Pos Diff |
SEO |
10 |
14 |
1582 |
1627 |
-45 |
0.63% |
0.86% |
-0.23% |
6.2 |
6.2 |
0 |
|
6 |
4 |
911 |
1177 |
-266 |
0.66% |
0.34% |
0.32% |
11 |
11 |
0 |
adwords certification |
11 |
5 |
986 |
1047 |
-61 |
1.12% |
0.48% |
0.64% |
13 |
13 |
0 |
hubspot certification |
5 |
2 |
641 |
765 |
-124 |
0.78% |
0.26% |
0.52% |
8.8 |
8.8 |
0 |
business analyst course |
17 |
25 |
632 |
735 |
-103 |
2.69% |
3.40% |
-0.71% |
13 |
13 |
0 |
certified business analysis professional |
4 |
4 |
354 |
337 |
17 |
1.13% |
1.19% |
-0.06% |
9.3 |
9.3 |
0 |
Analytics certification pune |
9 |
10 |
207 |
249 |
-42 |
4.35% |
4.02% |
0.33% |
3.5 |
3.5 |
0 |
change management training |
2 |
3 |
246 |
235 |
11 |
0.81% |
1.28% |
-0.47% |
29 |
29 |
0 |
soft skill training |
2 |
6 |
188 |
220 |
-32 |
1.06% |
2.73% |
-1.67% |
12 |
12 |
0 |
Analytics certification bangalore |
12 |
12 |
231 |
210 |
21 |
5.19% |
5.71% |
-0.52% |
3.7 |
3.7 |
0 |
big data training in bangalore |
1 |
2 |
123 |
196 |
-73 |
0.81% |
1.02% |
-0.21% |
10 |
10 |
0 |
financial modelling course |
5 |
4 |
133 |
169 |
-36 |
3.76% |
2.37% |
1.39% |
17 |
17 |
0 |
Analytics certification chennai |
12 |
15 |
196 |
166 |
30 |
6.12% |
9.04% |
-2.92% |
2.9 |
2.9 |
0 |
hubspot foundation certification |
6 |
23 |
101 |
149 |
-48 |
5.94% |
15.44% |
-9.50% |
3 |
3 |
0 |
frm course |
4 |
3 |
162 |
147 |
15 |
2.47% |
2.04% |
0.43% |
17 |
17 |
0 |
pmi acp training |
4 |
3 |
103 |
113 |
-10 |
3.88% |
2.65% |
1.23% |
19 |
19 |
0 |
From the above date we understand the seasonal drop is literal and the drop in average position by 3% could have been for some reason and meanwhile there will be an impact of that Avg position drop which we will be figuring out in the other three reasons of drop in organic traffic.
Case 2: Improvement in the Average position and drop in traffic.
This is the case when the CTR is low and this usually happens for the following reasons:
- Illegibly written titles and descriptions
- Product availability as discussed above
- The brand value
- External factors such as paid marketing and advertisements can consume the traffic.
Queries |
Week2 Clicks |
Week1 Clicks |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
Week2 Avg Pos |
Week1 Avg Pos |
Avp Pos Diff |
search engine optimization |
34 |
41 |
2904 |
3392 |
-488 |
1.17% |
1.21% |
-0.04% |
10 |
11 |
-1 |
SEO certification |
22 |
20 |
1799 |
2047 |
-248 |
1.22% |
0.98% |
0.24% |
7.8 |
7.9 |
-0.1 |
Adwords expert |
10 |
13 |
738 |
876 |
-138 |
1.36% |
1.48% |
-0.12% |
7.8 |
10 |
-2.2 |
Analytics |
3 |
3 |
918 |
867 |
51 |
0.33% |
0.35% |
-0.02% |
9.3 |
11 |
-1.7 |
Analytics certification cost |
5 |
9 |
538 |
640 |
-102 |
0.93% |
1.41% |
-0.48% |
9.6 |
9.8 |
-0.2 |
six sigma certification |
7 |
4 |
390 |
509 |
-119 |
1.79% |
0.79% |
1.00% |
10 |
11 |
-1 |
cspo |
12 |
10 |
450 |
507 |
-57 |
2.67% |
1.97% |
0.70% |
6.9 |
7.3 |
-0.4 |
Analytics training centre cochin |
0 |
1 |
0 |
448 |
-448 |
0% |
0.22% |
-0.22% |
0 |
2.3 |
-2.3 |
agile certification |
0 |
3 |
0 |
438 |
-438 |
0% |
0.68% |
-0.68% |
0 |
11 |
-11 |
project management certification |
4 |
11 |
383 |
430 |
-47 |
1.04% |
2.56% |
-1.52% |
13 |
15 |
-2 |
hubspot training |
13 |
7 |
462 |
388 |
74 |
2.81% |
1.80% |
1.01% |
39 |
45 |
-6 |
itil foundation certification |
9 |
4 |
452 |
387 |
65 |
1.99% |
1.03% |
0.96% |
23 |
26 |
-3 |
digital marketing course |
7 |
2 |
435 |
360 |
75 |
1.61% |
0.56% |
1.05% |
41 |
46 |
-5 |
simpliearn pune |
16 |
12 |
314 |
343 |
-29 |
5.10% |
3.50% |
1.60% |
4.7 |
4.8 |
-0.1 |
scrum master training |
2 |
7 |
282 |
329 |
-47 |
0.71% |
2.13% |
-1.42% |
22 |
27 |
-5 |
agile search engine optimization |
8 |
6 |
311 |
323 |
-12 |
2.57% |
1.86% |
0.71% |
9.5 |
9.6 |
-0.1 |
simpliearn solutions |
15 |
11 |
227 |
281 |
-54 |
6.61% |
3.91% |
2.70% |
1.3 |
1.6 |
-0.3 |
safe training |
11 |
10 |
256 |
259 |
-3 |
4.30% |
3.86% |
0.44% |
9.2 |
9.5 |
-0.3 |
pmi acp |
3 |
4 |
237 |
244 |
-7 |
1.27% |
1.64% |
-0.37% |
8.5 |
8.7 |
-0.2 |
safe agilist |
3 |
2 |
222 |
242 |
-20 |
1.35% |
0.83% |
0.52% |
9 |
9.3 |
-0.3 |
adwords foundation |
4 |
3 |
250 |
231 |
19 |
1.60% |
1.30% |
0.30% |
16 |
18 |
-2 |
Case3: Drop in traffic due to drop in the average positions of keywords
Queries |
Week2 Clicks |
Week1 Clicks |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
Week2 Avg Pos |
Week1 Avg Pos |
Avp Pos Diff |
Analytics certification |
68 |
78 |
3200 |
3589 |
-389 |
2.12% |
2.17% |
-0.05% |
7.8 |
7.5 |
0.3 |
project management courses |
49 |
61 |
768 |
749 |
19 |
6.38% |
8.14% |
-1.76% |
6.3 |
5.6 |
0.7 |
project management course |
23 |
19 |
209 |
242 |
-33 |
11% |
7.85% |
3.15% |
4.8 |
4.2 |
0.6 |
Analytics training |
18 |
25 |
358 |
507 |
-149 |
5.03% |
4.93% |
0.10% |
7.5 |
6.6 |
0.9 |
safe agile certification |
17 |
19 |
704 |
800 |
-96 |
2.41% |
2.38% |
0.03% |
8.6 |
8.3 |
0.3 |
Analytics certification cost in chennai |
13 |
9 |
71 |
42 |
29 |
18.31% |
21.43% |
-3.12% |
2.8 |
2.5 |
0.3 |
safe certification |
11 |
14 |
549 |
578 |
-29 |
2% |
2.42% |
-0.42% |
7.4 |
7.1 |
0.3 |
project management courses in mumbai |
11 |
3 |
62 |
50 |
12 |
17.74% |
6% |
11.74% |
1.9 |
1.8 |
0.1 |
primavera p6 training |
10 |
11 |
114 |
145 |
-31 |
8.77% |
7.59% |
1.18% |
13 |
11 |
2 |
pgmp certification training |
10 |
4 |
60 |
65 |
-5 |
16.67% |
6.15% |
10.52% |
3.9 |
3.4 |
0.5 |
Email certification |
10 |
9 |
757 |
785 |
-28 |
1.32% |
1.15% |
0.17% |
9 |
8.9 |
0.1 |
agile training |
10 |
18 |
529 |
692 |
-163 |
1.89% |
2.60% |
-0.71% |
20 |
17 |
3 |
Analytics course |
9 |
14 |
425 |
485 |
-60 |
2.12% |
2.89% |
-0.77% |
22 |
21 |
1 |
project management courses in bangalore |
8 |
6 |
67 |
51 |
16 |
11.94% |
11.76% |
0.18% |
2.4 |
1.8 |
0.6 |
Analytics certification in delhi |
8 |
16 |
65 |
86 |
-21 |
12.31% |
18.60% |
-6.29% |
2.2 |
2.1 |
0.1 |
Email training |
8 |
11 |
268 |
273 |
-5 |
2.99% |
4.03% |
-1.04% |
4.7 |
4.4 |
0.3 |
This case as well is true, there is a drop in the position and hence there is a drop in CTR and eventually a drop in traffic.
No, getting to the logic of what causes a keyword position drop?
There is one single metric that’s the root cause of the drop in SERP and that’s due to the drop in crawl stats such as crawl frequency and crawl rate,
But how can this Crawl rate or Crawl frequency come down?
- When the load times are good?
- When the file’s sizes are small?
- When the servers are good?
- When there is no change in the backlink profile?
These are not the only problems that impact the crawl stats, there are many more and they are as follows:
- Server errors- Header status 500:
This is one of the common issues when the hosting is on shared servers.
The 500 status header will be served when the server is busy and not be able to handle the request and this happens for the following reasons,
- When there is an enormous traffic to the website, as a result, the servers won’t be able to handle a large number of demanding requests and that when they show a 500 server error either to the user or to the bots or to both.
- Lots of bot traffic to the server and servers won’t be able to handle the requests and in turn, shows 500 server error.
- There is a possibility of the confirmation files going wrong at a moment or sometimes the commands handling the instructions itself might go wrong.
Some noticeable dip in the horizontal lines exactly the day of Independence day, and it’s not a sharp dip, it’s actually linear over that week. Let’s go a bit deeper to that date, I also observed the Avg position of keywords has come down by 3% and this would have been the actual reason for the drop.
Week |
Impressions |
Clicks/Traffic |
CTR |
Avg Pos |
Week1 Aug |
319539 |
9463 |
2.96% |
24 |
Week2 Aug |
283033 |
8541 |
3.02% |
24.6 |
Diff |
-36506 |
-922 |
0.0006 |
0.6 |
%Change |
–11% |
-10% |
2% |
3% |
2. Drop in Traffic due to the change in keywords SERP and CTR:
Case1: No Change in Avg position of keywords but a gradual drop in impressions:
This clearly implies that the number of searches has come down from the region which is prominently contributing for impressions, here, in this case, India is the major contributor and hence there is a seasonal impact.
For Data Privacy reasons the numbers are alerted in the table and across the article, the altered number has a proportionate difference across the complete set and dimensions,
Queries |
Week2 Clicks |
Week1 Clicks |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
Week2 Avg Pos |
Week1 Avg Pos |
Avp Pos Diff |
SEO |
10 |
14 |
1582 |
1627 |
-45 |
0.63% |
0.86% |
-0.23% |
6.2 |
6.2 |
0 |
|
6 |
4 |
911 |
1177 |
-266 |
0.66% |
0.34% |
0.32% |
11 |
11 |
0 |
adwords certification |
11 |
5 |
986 |
1047 |
-61 |
1.12% |
0.48% |
0.64% |
13 |
13 |
0 |
hubspot certification |
5 |
2 |
641 |
765 |
-124 |
0.78% |
0.26% |
0.52% |
8.8 |
8.8 |
0 |
business analyst course |
17 |
25 |
632 |
735 |
-103 |
2.69% |
3.40% |
-0.71% |
13 |
13 |
0 |
certified business analysis professional |
4 |
4 |
354 |
337 |
17 |
1.13% |
1.19% |
-0.06% |
9.3 |
9.3 |
0 |
Analytics certification pune |
9 |
10 |
207 |
249 |
-42 |
4.35% |
4.02% |
0.33% |
3.5 |
3.5 |
0 |
change management training |
2 |
3 |
246 |
235 |
11 |
0.81% |
1.28% |
-0.47% |
29 |
29 |
0 |
soft skill training |
2 |
6 |
188 |
220 |
-32 |
1.06% |
2.73% |
-1.67% |
12 |
12 |
0 |
Analytics certification bangalore |
12 |
12 |
231 |
210 |
21 |
5.19% |
5.71% |
-0.52% |
3.7 |
3.7 |
0 |
big data training in bangalore |
1 |
2 |
123 |
196 |
-73 |
0.81% |
1.02% |
-0.21% |
10 |
10 |
0 |
financial modelling course |
5 |
4 |
133 |
169 |
-36 |
3.76% |
2.37% |
1.39% |
17 |
17 |
0 |
Analytics certification chennai |
12 |
15 |
196 |
166 |
30 |
6.12% |
9.04% |
-2.92% |
2.9 |
2.9 |
0 |
hubspot foundation certification |
6 |
23 |
101 |
149 |
-48 |
5.94% |
15.44% |
-9.50% |
3 |
3 |
0 |
frm course |
4 |
3 |
162 |
147 |
15 |
2.47% |
2.04% |
0.43% |
17 |
17 |
0 |
pmi acp training |
4 |
3 |
103 |
113 |
-10 |
3.88% |
2.65% |
1.23% |
19 |
19 |
0 |
From the above date we understand the seasonal drop is literal and the drop in average position by 3% could have been for some reason and meanwhile there will be an impact of that Avg position drop which we will be figuring out in the other three reasons of drop in organic traffic.
Case 2: Improvement in the Average position and drop in traffic.
This is the case when the CTR is low and this usually happens for the following reasons:
- Illegibly written titles and descriptions
- Product availability as discussed above
- The brand value
- External factors such as paid marketing and advertisements can consume the traffic.
Queries |
Week2 Clicks |
Week1 Clicks |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
Week2 Avg Pos |
Week1 Avg Pos |
Avp Pos Diff |
search engine optimization |
34 |
41 |
2904 |
3392 |
-488 |
1.17% |
1.21% |
-0.04% |
10 |
11 |
-1 |
SEO certification |
22 |
20 |
1799 |
2047 |
-248 |
1.22% |
0.98% |
0.24% |
7.8 |
7.9 |
-0.1 |
Adwords expert |
10 |
13 |
738 |
876 |
-138 |
1.36% |
1.48% |
-0.12% |
7.8 |
10 |
-2.2 |
Analytics |
3 |
3 |
918 |
867 |
51 |
0.33% |
0.35% |
-0.02% |
9.3 |
11 |
-1.7 |
Analytics certification cost |
5 |
9 |
538 |
640 |
-102 |
0.93% |
1.41% |
-0.48% |
9.6 |
9.8 |
-0.2 |
six sigma certification |
7 |
4 |
390 |
509 |
-119 |
1.79% |
0.79% |
1.00% |
10 |
11 |
-1 |
cspo |
12 |
10 |
450 |
507 |
-57 |
2.67% |
1.97% |
0.70% |
6.9 |
7.3 |
-0.4 |
Analytics training centre cochin |
0 |
1 |
0 |
448 |
-448 |
0% |
0.22% |
-0.22% |
0 |
2.3 |
-2.3 |
agile certification |
0 |
3 |
0 |
438 |
-438 |
0% |
0.68% |
-0.68% |
0 |
11 |
-11 |
project management certification |
4 |
11 |
383 |
430 |
-47 |
1.04% |
2.56% |
-1.52% |
13 |
15 |
-2 |
hubspot training |
13 |
7 |
462 |
388 |
74 |
2.81% |
1.80% |
1.01% |
39 |
45 |
-6 |
itil foundation certification |
9 |
4 |
452 |
387 |
65 |
1.99% |
1.03% |
0.96% |
23 |
26 |
-3 |
digital marketing course |
7 |
2 |
435 |
360 |
75 |
1.61% |
0.56% |
1.05% |
41 |
46 |
-5 |
simpliearn pune |
16 |
12 |
314 |
343 |
-29 |
5.10% |
3.50% |
1.60% |
4.7 |
4.8 |
-0.1 |
scrum master training |
2 |
7 |
282 |
329 |
-47 |
0.71% |
2.13% |
-1.42% |
22 |
27 |
-5 |
agile search engine optimization |
8 |
6 |
311 |
323 |
-12 |
2.57% |
1.86% |
0.71% |
9.5 |
9.6 |
-0.1 |
simpliearn solutions |
15 |
11 |
227 |
281 |
-54 |
6.61% |
3.91% |
2.70% |
1.3 |
1.6 |
-0.3 |
safe training |
11 |
10 |
256 |
259 |
-3 |
4.30% |
3.86% |
0.44% |
9.2 |
9.5 |
-0.3 |
pmi acp |
3 |
4 |
237 |
244 |
-7 |
1.27% |
1.64% |
-0.37% |
8.5 |
8.7 |
-0.2 |
safe agilist |
3 |
2 |
222 |
242 |
-20 |
1.35% |
0.83% |
0.52% |
9 |
9.3 |
-0.3 |
adwords foundation |
4 |
3 |
250 |
231 |
19 |
1.60% |
1.30% |
0.30% |
16 |
18 |
-2 |
Case3: Drop in traffic due to drop in the average positions of keywords
Queries |
Week2 Clicks |
Week1 Clicks |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
Week2 Avg Pos |
Week1 Avg Pos |
Avp Pos Diff |
Analytics certification |
68 |
78 |
3200 |
3589 |
-389 |
2.12% |
2.17% |
-0.05% |
7.8 |
7.5 |
0.3 |
project management courses |
49 |
61 |
768 |
749 |
19 |
6.38% |
8.14% |
-1.76% |
6.3 |
5.6 |
0.7 |
project management course |
23 |
19 |
209 |
242 |
-33 |
11% |
7.85% |
3.15% |
4.8 |
4.2 |
0.6 |
Analytics training |
18 |
25 |
358 |
507 |
-149 |
5.03% |
4.93% |
0.10% |
7.5 |
6.6 |
0.9 |
safe agile certification |
17 |
19 |
704 |
800 |
-96 |
2.41% |
2.38% |
0.03% |
8.6 |
8.3 |
0.3 |
Analytics certification cost in chennai |
13 |
9 |
71 |
42 |
29 |
18.31% |
21.43% |
-3.12% |
2.8 |
2.5 |
0.3 |
safe certification |
11 |
14 |
549 |
578 |
-29 |
2% |
2.42% |
-0.42% |
7.4 |
7.1 |
0.3 |
project management courses in mumbai |
11 |
3 |
62 |
50 |
12 |
17.74% |
6% |
11.74% |
1.9 |
1.8 |
0.1 |
primavera p6 training |
10 |
11 |
114 |
145 |
-31 |
8.77% |
7.59% |
1.18% |
13 |
11 |
2 |
pgmp certification training |
10 |
4 |
60 |
65 |
-5 |
16.67% |
6.15% |
10.52% |
3.9 |
3.4 |
0.5 |
Email certification |
10 |
9 |
757 |
785 |
-28 |
1.32% |
1.15% |
0.17% |
9 |
8.9 |
0.1 |
agile training |
10 |
18 |
529 |
692 |
-163 |
1.89% |
2.60% |
-0.71% |
20 |
17 |
3 |
Analytics course |
9 |
14 |
425 |
485 |
-60 |
2.12% |
2.89% |
-0.77% |
22 |
21 |
1 |
project management courses in bangalore |
8 |
6 |
67 |
51 |
16 |
11.94% |
11.76% |
0.18% |
2.4 |
1.8 |
0.6 |
Analytics certification in delhi |
8 |
16 |
65 |
86 |
-21 |
12.31% |
18.60% |
-6.29% |
2.2 |
2.1 |
0.1 |
Email training |
8 |
11 |
268 |
273 |
-5 |
2.99% |
4.03% |
-1.04% |
4.7 |
4.4 |
0.3 |
This case as well is true, there is a drop in the position and hence there is a drop in CTR and eventually a drop in traffic.
No, getting to the logic of what causes a keyword position drop?
There is one single metric that’s the root cause of the drop in SERP and that’s due to the drop in crawl stats such as crawl frequency and crawl rate,
But how can this Crawl rate or Crawl frequency come down?
- When the load times are good?
- When the file’s sizes are small?
- When the servers are good?
- When there is no change in the backlink profile?
These are not the only problems that impact the crawl stats, there are many more and they are as follows:
- Server errors- Header status 500:
This is one of the common issues when the hosting is on shared servers.
The 500 status header will be served when the server is busy and not be able to handle the request and this happens for the following reasons,
- When there is an enormous traffic to the website, as a result, the servers won’t be able to handle a large number of demanding requests and that when they show a 500 server error either to the user or to the bots or to both.
- Lots of bot traffic to the server and servers won’t be able to handle the requests and in turn, shows 500 server error.
- There is a possibility of the confirmation files going wrong at a moment or sometimes the commands handling the instructions itself might go wrong.
- Soft 404 errors:
Soft 404 errors are those cases when the bots find a status code other than 404 for the non-existing
file on the server. In simple words, if the requested file is not on the server, a server is supposed to serve a 404
(Not found) message/status header and instead if it’s servers serve some other status header instead of
404, it will be considered to be as the soft 404 error, this would impact the crawl rate of the website at a significant level.
- Access denied- Status header 407:
Servers are denying the bots crawling the content and this usually happens when,
- Users compulsorily need to login to access content, there will not be an issue if the actual intended pages are denying the access, but if in case the organic performing pages denies access then this can be considered as a huge problem.
- Server setting may need proxy authentication and in that case, servers would block the Google bot that is crawling the website.
- In case if any of the valid content or URLs are blocked under robots then the” access denied” error is a tremendous error to be handled on priority.
Just to figure out what actually went wrong I cross-checked against the three reasons I found,
Everything is perfectly alright against the points 1st and 3rd,and I could only suspect the new implementation done in the recent times,
The issue is with the rounded content, the recently updated privacy policy could be the reason for it as everything else remains the same including the server setting, the same can be seen in the first image of the server denied section as the errors appeared only in the recent times
- Not Found errors- Status header 404:
As seen in the above image the errors have increased by a significant number in the recent times and these 404 errors will impact the crawl rate to a significant level and in turn impacting the avg positions of the keywords on SERP.
3. Drop in traffic due to paid marketing:
Drop in traffic that’s coming through the brand terms, and major revenue driving terms as paid marketers bid on it and take some of the shares,
Here are some steps in understanding how to find it,
Brand Organic Search Traffic Drop with the paid marketing:
In the recent times we have launched paid campaigns, as some of the competitors were bidding on our brand terms and were taking off most of the converting traffic from us and ergo we considered launching the paid campaigns:
I have also gone through the brand keywords of the webmaster data and I was able to figure that there is a drop in CTR of many of the traffic driving keywords, the keyword data with a drop in CTR can be found below.
Queries |
Week2 Clicks |
Week1 Clicks |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
simpliearn |
2160 |
2435 |
32383 |
33729 |
-1346 |
6.67% |
7.22% |
-0.55% |
simpli earn |
655 |
809 |
10138 |
10803 |
-665 |
6.46% |
7.49% |
-1.03% |
simpliearn SEO |
68 |
111 |
1726 |
1696 |
30 |
3.94% |
6.54% |
-2.60% |
simpli earns |
54 |
59 |
851 |
907 |
-56 |
6.35% |
6.50% |
-0.15% |
simpliearn pune |
52 |
36 |
1213 |
717 |
496 |
4.29% |
5.02% |
-0.73% |
simpliearn solutions |
50 |
76 |
932 |
1261 |
-329 |
5.36% |
6.03% |
-0.67% |
simpliearn solutions pvt ltd |
34 |
33 |
613 |
534 |
79 |
5.55% |
6.18% |
-0.63% |
simpliearn bangalore |
32 |
41 |
1028 |
875 |
153 |
3.11% |
4.69% |
-1.58% |
simpliearn solutions pvt. ltd |
27 |
26 |
419 |
341 |
78 |
6.44% |
7.62% |
-1.18% |
simpliearn solutions private limited |
25 |
17 |
396 |
271 |
125 |
6.31% |
6.27% |
0.04% |
simpliearn solutions pvt. ltd. |
25 |
28 |
324 |
282 |
42 |
7.72% |
9.93% |
-2.21% |
simpliearn reviews |
25 |
45 |
485 |
558 |
-73 |
5.15% |
8.06% |
-2.91% |
simpliearns |
24 |
30 |
343 |
356 |
-13 |
7% |
8.43% |
-1.43% |
simpliearn careers |
22 |
30 |
70 |
71 |
-1 |
31.43% |
42.25% |
-10.82% |
Paid Campaigns effecting the Non Brand Search Traffic (Hit on CTR due to paid search
marketing)
To understand this let’s consider some of the products/services that satisfy the following conditions,
- The course which has active paid marketing campaigns
- The search volume has no change
- The Avg position has no change
- All the competitors ranking and brand value remain same
- Our brand value to is same
- There are no changes in any of the external factors except the paid marketing
Now consider all the above, in absence of paid marketing the CTR’s and traffic coming from the specific keywords should be same, but then there is drop in traffic despite no change or improvement in the average position of the keyword on the SERP as depicted in the below table,
Queries |
Week2 Clicks |
Week1 Clicks |
Clicks Diff |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Week2 CTR |
Week1 CTR |
CTR Diff |
Week2 Avg Pos |
Week1 Avg Pos |
Avp Pos Diff |
project management courses |
49 |
61 |
-12 |
768 |
749 |
19 |
6.38% |
8.14% |
-1.76% |
6.3 |
5.6 |
0.7 |
search engine optimization bangalore |
4 |
8 |
-4 |
90 |
88 |
2 |
4.44% |
9.09% |
-4.65% |
3.7 |
3.5 |
0.2 |
digital marketing course berlin |
3 |
4 |
-1 |
9 |
8 |
1 |
33.33% |
50% |
-16.67% |
1.1 |
1 |
0.1 |
safe agile certification chennai |
3 |
4 |
-1 |
9 |
8 |
1 |
33.33% |
50% |
-16.67% |
2.1 |
2 |
0.1 |
Analytics certification training |
2 |
3 |
-1 |
148 |
136 |
12 |
1.35% |
2.21% |
-0.86% |
26 |
24 |
2 |
safe training in pune |
2 |
3 |
-1 |
6 |
5 |
1 |
33.33% |
60% |
-26.67% |
1.3 |
1 |
0.3 |
quality management certification courses |
2 |
3 |
-1 |
48 |
36 |
12 |
4.17% |
8.33% |
-4.16% |
6.4 |
6.1 |
0.3 |
scaled agile certification |
2 |
7 |
-5 |
184 |
183 |
1 |
1.09% |
3.83% |
-2.74% |
8 |
7.8 |
0.2 |
p6 primavera training |
1 |
2 |
-1 |
19 |
17 |
2 |
5.26% |
11.76% |
-6.50% |
8.5 |
8.2 |
0.3 |
business analyst course in hyderabad |
1 |
2 |
-1 |
51 |
44 |
7 |
1.96% |
4.55% |
-2.59% |
8.5 |
7.2 |
1.3 |
online safe training |
1 |
2 |
-1 |
7 |
5 |
2 |
14.29% |
40% |
-25.71% |
3.3 |
3.2 |
0.1 |
scrum master training dallas |
1 |
3 |
-2 |
6 |
5 |
1 |
16.67% |
60% |
-43.33% |
2.7 |
2.4 |
0.3 |
leading safe |
1 |
4 |
-3 |
208 |
207 |
1 |
0.48% |
1.93% |
-1.45% |
8.8 |
8.7 |
0.1 |
Analytics certification hyderabad |
1 |
6 |
-5 |
53 |
45 |
8 |
1.89% |
13.33% |
-11.44% |
2.3 |
2 |
0.3 |
agile search engine optimization in chennai |
1 |
3 |
-2 |
21 |
15 |
6 |
4.76% |
20% |
-15.24% |
2.6 |
1.4 |
1.2 |
togaf training |
1 |
2 |
-1 |
101 |
90 |
11 |
0.99% |
2.22% |
-1.23% |
51 |
37 |
14 |
4. Any technical glitches/blunders either by the marketing teams or development or both on the system.
This is one of the most common issues we come across and impacting greatly but least noticed. There is something similar that occurred with our website as well, there were around 2000 web pages which got blocked for search engines crawl and index due to the improper
communication between the technical team and the marketing team and this has led to a drop in traffic coming from those web pages,
Below is the table which clearly depicts the above statement,
Queries |
Week 2 Imp |
Week1 Imp |
Imp Diff |
Analytics training centre cochin |
0 |
448 |
-448 |
agile certification |
0 |
438 |
-438 |
agile project management training |
0 |
230 |
-230 |
Adwords expert training |
0 |
225 |
-225 |
big data hadoop training |
0 |
209 |
-209 |
business analyst training |
0 |
207 |
-207 |
cissp training |
0 |
199 |
-199 |
digital marketing course in indore |
0 |
172 |
-172 |
project management institute |
0 |
168 |
-168 |
iiba |
0 |
162 |
-162 |
digital marketing course in chandigarh |
0 |
151 |
-151 |
pgmp |
0 |
150 |
-150 |
hubspot training in chennai |
0 |
139 |
-139 |
digital marketing training in indore |
0 |
137 |
-137 |
six sigma course |
0 |
132 |
-132 |
safe agilist certification |
0 |
130 |
-130 |
agile courses |
0 |
118 |
-118 |
business management training courses |
0 |
116 |
-116 |
simpliearn address |
0 |
113 |
-113 |
certified scrum professional |
0 |
112 |
-112 |
togaf 8 certification,   on page seo activities list,   track serp position,   google search remove site from results,   average organic ctr,   that’s all we know.,   check competitor backlinks,   digital marketing salary,   adwords course indore,   search specific website,   digital marketing course free,   seo juice,   iiba chennai,   google search term frequency tool,   conversion optimisation,   spectrum going all digital,