By and large, the largest obstacle to getting out of a penalty tends to be the website owner him or herself. I do not mean to trivialize what webmasters go through, I often think it is incredibly unfair, but we all fall victim to a human tendency to prefer avoiding loss than acquiring gain. This “loss aversion” has set webmasters up to fail time and time again once a penalty has set in. So here are a few of the biggest mistakes webmasters make…

Guessing Google’s Link Qualification Algorithm

First, let me say that there are some excellent tools out there for link qualification. Cemper’s Link Detox, the new Link Risk and Remove ‘ems internal link qualification system all do decent jobs of identifying which links are most concerning. In fact, I can say in no uncertain terms that Remove ‘ems system is the least “specific” of these 3 in terms of identifying exactly which links are the worst, and instead errs on the side of removals. The reason is simple…

Missing 1 bad link can mean another round of reconsideration requests

Webmasters regularly make the error of trying to to remove the fewest number of links possible to get back into search results. The logic behind this is sound. Webmasters don’t want to get rid of any links that still pass value. However, just like the doctor who chooses to surgically remove a generous portion of tissue around a melanoma, so should you be prepared to remove a generous number of links around those you are certain are toxic to make sure you limit the number of necessary reconsideration requests.

More importantly, because the removal rate percentage varies greatly, it is better to reach out to a large number of potential removal opportunities early on rather than find out your first pass only produced a 5% success rate. The last thing you want to do is go through 3 or 4 reconsideration requests because each time you haven’t quite removed enough.

Pay the Piper

So many in the search industry have indicated that you should not to pay webmasters to remove bad links and I could not disagree with them more. There are several excellent reasons to pay “bounties” for link removals…

1. removing links takes actual effort, and a webmaster should be compensated for it
2. if you spammed the link, you should cover both their effort and their hardship
3. paying to remove links speeds up the process. Nothing motivates a removal faster than a quick buck.

Just remember that each link you leave up is a potential extra 2 weeks waiting for that next reconsideration request to come in. Can your business survive that easier than paying another $15 bounty?

Sitting on Your Hands

This one is incredibly common and, frankly, infuriating. I can’t begin to count the number of times I have heard these words “we filed a reconsideration request and now we are just waiting.” Are you crazy? If your boat is sinking do you stop trying to empty the water out because you put in a call to the Coast Guard? Get back to work! If you used the Disavow tool, you have more work to do. Keep working on that list so the day they respond, if it is negative, you can follow up with evidence that – in good faith – you continued to work and you now have a new set of link removals to show them. If you aren’t prepared with a quick response, it is evidence that you are trying to get by doing as little as possible.

So, time to fight.

There is a light at the end of the tunnel, but you aren’t going to get there by complaining or waiting. Grow up, accept what things you have done or have been done in your name, own up to them, clean the mess and get back to business. It is hard as hell, no doubt, don’t think for a second you have no control.

However, what I do mean to say, with great emphasis, is that SEO as a specialty has, does, will and should continue to exist.

I felt like I should chime in regarding Rand’s excellent Whiteboard Friday today.

1. SEO is Bigger than SEO:
SEO or “Search Engine Optimization” is is a statement of purpose, not a statement of methods. Carl Sagan once said, “If you wish to make an apple pie from scratch, you must first invent the universe.” I suppose we could call bakers Gods if this were the case, but I believe most of us would agree that to be generally false. Except for the creators of the bourbon pecan cupcake‘. If your goal is to bake foods, you are a baker, regardless of the tactics necessary to bake those foods. Listing off CRO and UX and Branding as necessary components of a successful SEO campaign does not change the intent or purpose of that campaign – to increase qualified organic traffic from search engines. While the responsibilities have grown, the goal has not. If the goal has shifted, then yes, you should change your title. If tomorrow, Google changes all the necessary tactics to rank, I will still be an SEO. But would I still be an Inbound Marketer?

2. Perception is Bad:
I agree that the perception is generally bad, but this obstacle does not change what we do nor should we surrender our true identities to a new facade because we can’t shake the vestiges of our past. Using terms like “Inbound Marketing” and “Online Marketer” merely obfuscate the reality of our primary goals in the hopes that we aren’t lumped together with people’s misguided perceptions.

3. We are Selling Ourselves Short:
I believe that Rand’s arguments here are what I find most disconcerting. SEO is getting harder and Google intends to make it harder and harder. When we begin to call ourselves inbound marketers and begin devote more and more of our time to ancillary marketing tactics that have tangential impacts on organic search, we have moved our eyes off the ball. We have lost focus. Now, I will admit, that focus is a luxury that only some businesses can afford – those that can afford a team of individuals who can have multiple marketers working on a cohesive strategy but with individuals focusing on different aspects.

Final thoughts.
I do not mean to say that people who are inbound marketers out there should not call themselves inbound marketers. They absolutely should. I also do not mean to say that a group of SEOs are shifting away from SEO to inbound marketing. They absolutely are. However, what I do mean to say, with great emphasis, is that SEO as a specialty has, does, will and should continue to exist. That specialty will likely become more difficult to master, more costly to perform, and will require more skills than before – but I will gladly be a part of it.

I have a purpose. It is to increase qualified organic traffic from search engines. I embody that purpose. I am an SEO.

Biomagnification is a fairly simple principle that through a series of prey-predator relationships, toxic substances tend to accumulate in higher percentages among organisms higher in the food chain. You can see a visualization of this in the image to the left. As mercury accumulates in various organisms, predators consume those organisms and absorb those toxins. Unless the organism has a way of disposing those toxins, they can and will magnify in their accumulation.

This is the same when merging sites with one another. If you have many micro-sites or subdomains that have been received less-than-kosher links in the past, you may have been able to dodge some algorithm updates like Penguin simply because the sites’ link profiles were too small. Your rankings might not be that great either on these sites, but you have no evidence of a link penalty. However, redirecting multiple sites to another presents a potential for biomagnification because the combined link profiles may be enough to trigger penalties, filters or algorithmic devaluations.

The most interesting implication is this: even if your main site and all the sites/subdomains you intend to redirect to your main site appear to have no link penalties (manual or algorithmic), the combination of those sites could trigger one.

Of course, the simple solution is to be thoughtful when joining sites together. Determine what new link thresholds will be once you combine anchor text, root link domains, etc. to make sure you aren’t stepping over any lines. And, of course, if you are – remove the concerning links before doing the redirect to avoid the wrath of search engines.

1. Since its inception back in October of 2012, brokenlinkbuilding.com has discovered nearly 10,000,000 broken link opportunities. This is pretty amazing. The tool, which is still not nearly as popular as it should be, has managed to discover millions of straight-forward, link building opportunities.
    
2. 8269448 of those backlink opportunities come from 404 pages with over 100 backlinks. This is huge in terms of conversion opportunities, because it means that for each single resource you recreate, you have an opportunity to go after hundreds of links.
    
3. There are over 14,000 404 pages in the system with more than 100 unique linking domains.
    
4. 142,088 backlink opportunities with 100+ links and an onTopic score of B and 29651 with a score of A – meaning 99% relevant, which means…
    
5. On average, every single keyword returns at least 1 highly relevant, 100+ unique linking domain opportunity…
    
6. Which means that, even if you have a meager 2% conversion rate on your campaigns, you just got 2 permanent, natural links for $6.70.
    

These numbers are astounding for a product that is, frankly, still brand new. The reason why the numbers are so shocking is that we have nearly 2 decades of broken resources built up. This is a veritable link building gold rush.

So What Does This Mean for Me

Honestly, I think everyone has 2 choices.

1. Start broken link building on your own, either following the steps in the BLBB or using a tool like Garrett’s
2. Or hire an agency to do it for you. Any agency worth a grain of salt should be in on this method, Virante surely is.

What isn’t an option is not doing broken link building – and here is why. Broken link building is only slightly a renewable resource. The best opportunities can and will be taken by someone in your niche. Time is of the essence.

If you are familiar with the SEOMoz, Majestic and AHRefs APIs, it literally takes only an hour or so to create your own widget that pulls the data directly into your Piwik analytics install.

1. It is not my data to give away. It took me a while to come to this conclusion. I always looked at Google Analytics as a trade off. I will let Google know about my site in exchange for a better way to visualize the usage on my site. But something kept nagging – and that was the reality that Google is not in the Analytics game for your site’s data. They are in the game for your users’ data. Chances are, your users have no idea that Google is collecting their usage data as they move through your site, and they definitely don’t understand the consequences of it. I think I deserve to know what my users are doing on my site, but do I deserve to be able to sell their data to a third party (in exchange for software)? That seems like a stretch.

2. It might help your competitor’s ad campaigns. If Google knows a user has been on your site, what is to keep them from retargeting advertisements for your competitors in the future based on that data point? I haven’t looked into this possibility at all, but I have a nagging suspicion it is one of the reasons why Google offers such a powerful tool set openly and freely.

3. Google Analytics doesn’t expose the raw data I hate having to go back to raw logs to try and understand exactly what is going on in a campaign.

4. Google Analytics is a closed environment If I want to tie rankings or links data into my analytics, I have to export the analytics data into either 3rd party software or a spreadsheet. I’d prefer to be able to improve the analytics program directly.

5. Information Parity As petty as this sounds, if Google isn’t going to give me the keywords from their site, why should I give them my entire site’s usage data? Why should I trust Google with my data when they won’t trust me with such a small amount of theirs? Why does Google think their users need to be protected from me, but for some reason I think my users don’t need to be protected from Google?

So, what is the replacement? Piwik

Piwik is a fairly robust open source analytics software. It is highly extensible which means I and my team can mod it to our hearts desire. Hopefully by this time next year we will have a unique, Piwik-powered SEO/Analytics hybrid which we can use on clients’ sites.

Over the past two weeks multiple respected bloggers in the search community have commented on the increasing abilities of Googlebot, especially following Google’s announcement that it can now handle some forms of AJAX. I have, admittedly, long believed that we over-estimate what Google and Googlebot are capable of, so I wanted to run a proper experiment to determine the exact capabilities of GoogleBot in reading and interpreting Javascript.

The Question

How sophisticated is Googlebot’s javascript interpretation and, more specifically, which Javascript functions can Google accurately interpret.

The Functions and Features Tested

• Simple Variables: Can Google Understand Simple Variable Assignment such as “var foo = ‘test content’; document.write(foo); “
• Simple Variable Concatenation: Can Google Interpret “var foo = ‘test content’; var foo += ‘ more ‘; document.write(foo); “
• Simple Document.write();
• Simple element.innerHTML();
• Dummy Variables: We added this test in to make sure Google only indexes data that is printed to the page, and not every string randomly stored in a variable.

The Methods Tested

• Inline: We tested javascript stored on the page
• Included: We tested javascript in a simple include
• Included behind Robots.txt: We tested javascript in an include blocked by Robots.txt

The Results

Hold Your Breath

Everyone now is probably staring at the Red highlighted column that indicates Googlebot can and will interpret Javascript hidden behind a robots.txt exclusion. I took the time to verify this and check it with multiple sources before finally reaching out to the source of truth. Was Googlebot really ignoring robots.txt when considering Javascript includes?

In short, no.

Best Practices for Robots.txt and Javascript

First, let me state that it is likely that Google will at some point (if they don’t already) use blocked .JS and .CSS as a negative signal. While there are legitimate reasons for this, there is no easy way for Google to verify that the contents of a page using these tactics are not greatly modified by the blocked files. So, be careful.

That being said, Matt was kind enough to respond in great detail to my findings, and pointed out several things one should consider when blocking .JS files which ultimately resulted in false positives in my analysis:

1. Give Your Robots.txt a Head Start: This makes a lot of sense, but most webmasters (myself included) handle the new content and robots.txt at the same time.
   

   “In an ideal world, you’d wait 12 hours just to be completely safe. Essentially, any time you make a new directory and block it at the same time, there’s a race condition where it’s possible we would fetch the test.js before we saw it was blocked in robots.txt. That’s what happened here.” – Matt Cutts

   It is certainly untenable for Googlebot to check the Robots.txt with every new file downloaded on your site, so giving that head start can make a big difference.

2. User-Agent Directives can Override One Another: This one was new to me, but it does make sense. If you begin with a generic “User-Agent: *” directive, and follow up with a specific directive, “User-Agent: Googlebot”, the latter overrides the former in terms of Googlebot, it does not append to it.
   

   If you disallow user-agent: * and then have a disallow user-agent: Googlebot, the more specific Googlebot section overrides the more general section–it doesn’t supplement it. – Matt Cutts

3. Robots.txt is only Respected Up to 500,000 Characters: I know this is a pretty big number, but if you have a lot of unique URLs to block, it can get messy. This is particularly frustrating with the Google Webmaster Tools Robots.txt checker, which only analyzes the first 100,000.
4. To Be Certain, Use the X-Robots-Tag: There is a great writeup here on how to use the HTTP Header X-Robots-Tag to indicate to Google that any file and filetype should not be indexed. Because this header is sent along with the file, Googlebot can respect it in real-time.
5. .JS Files can Be Slow to Clear from Index: As is the case with any lower-priority crawled document, .JS files can take a while to clear Google’s index if for some reason Google finds the blocked .JS.
   

   The crawl team said that once a .js file has been fetched, it can be cached in our indexing process for a while. – Matt Cutts

   This is certainly not an understatement. The .JS indexed 2 weeks ago is still present on pages that were indexed before Googlebot realized the exclusion. I believe, though, that you can always use the emergency removal tool if this happens.

Re-Running the Test

Of course, after hearing back from Matt, I needed to re-run the blocked .JS test to confirm. Sure enough, now that the .JS file was behind a previously-established blocked directory, Googlebot respected the disallow. (Also, just to be careful, I tested it on a separate domain with which Matt was not familiar, so I can assure you there was no trickery involved).

Take Aways

1. On Javascript: Google is actually interpreting the Javascript it spiders. It is not merely trying to extract strings of text and it does appear to be nuanced enough to know what text is and is not added to the Document Object Model. This is impressive.
2. On Experimenting: Confirm, retest, ask, retest, confirm, confirm, write, confirm, revise, confirm, publish.
3. On SEO: Learn new shit every day.

Update: Google has reopened the API for Raven Tools! Let’s keep a watch and see if they do the same thing for SEOMoz. Keep up the pressure!

What is Going Down?

If you are in the search marketing community, you are likely familiar with Raven Tools and SEOMoz. These two SEO tool leaders provide incredible tool sets for webmasters looking to improve their search performance – in both paid and organic. They aggregate data from numerous sources including the Google Adwords API. Back in September, SearchEngineLand.com reported that Google had begun a process of revoking a large number of AdWords users’ API access and then provisionally allow access. This explained why SEOMoz saw intermittent access of the API over the last several months. We have now learned that both SEOMoz and Raven Tools have lost their AdWords access.

Why is Google Doing This?

From Google’s perspective, the AdWords API is to be specifically used by AdWords customers to help improve their own campaigns. In general, they are not to be used as a part of an exposed public or private tool, even if the expressed use is to improve AdWords campaign performance. In reality, the AdWords API is one of Google’s biggest liabilities for numerous reasons. Here are just a few…

1. Traffic volume data provides competitors with information on target markets
2. Bid prices allow for advertisers to automatically compare prices and, potentially, determine better advertising opportunities
3. Bid prices allow competitor ad networks to price their products appropriately

Google’s Adwords API, despite being paid, is kept under incredibly strict usage guidelines that are enforced regularly and without much forgiveness. Why would Google create a paid service and then restrict the number of its customers and the volume of its usage so carefully? To protect industry dominance.

What Does the FTC Have to Do With This?

The FTC has been investigating Google for monopolistic practices for quite some time. While we can quibble back and forth about whether Google is genuinely a monopoly, there is little to no argument that Google is not trying to behave like one. The New York Times points out that “The investigators are… looking into whether Google’s automated advertising marketplace, AdWords, discriminates against advertisers from competing online commerce services like comparison shopping sites and consumer review Web sites.”

The tight reigns on the Adwords API do just this. You cannot know the cost of using AdWords without actually being a logged in customer of AdWords. The prices are private and restricted. It is a violation of their Terms and Conditions to extract data in any other way than those expressly permitted by Google (T&C Section 4(B)). This would be akin to WalMart refusing to publish their prices in any public fashion and then refuse to allow you to log-in and “extract” their prices in any way other than what they expressly tell you is OK so you know if you are getting a good deal – AND THEN STARTED ENFORCING IT by booting people out of the store who were caught writing down prices or seeing how many items were still in stock.

This kind of activity is alright when there are a bunch of other stores in the area, but when Google is by far the biggest advertising game in town, it starts to look very suspicious.

Hubris

The big word that comes to me here is hubris. Google is over-confident that the FTC is not willing to take them to court. Winning monopoly cases is expensive and right now the Federal Government doesn’t exactly have a huge budget. But this kind of activity simply laughs in the face of the FTC right as they are planning to release their findings. If I were Google, I would be more careful. But, alas, I am not. I’m just a guy trying to run a business.

Addendums…

Great feedback is coming in from the search community. Andrew Dumont mentioned that by not allowing users to purchase ads from within their app, they are in violation of the Terms & Conditions of the API. If I wasn’t clear above, I don’t mean to state that people are wrongfully having their API access revoked. The T&C are being fairly enforced. My intent was to note that these rigorously enforced T&C serve a very specific purpose of intentionally hiding both their pricing and their inventory to make it difficult for consumers to compare costs and opportunities with both paid and free alternatives.

The Deception of Integer Rankings

A key problem with much of SEO is that rank order is deceptive. We perceive sites as positions 1 through 10, each with a fixed integer position when, in reality, their proximity to one another in terms of actual relationship-to-the-query scores can vary greatly. It is this hidden, underlying score that is key to many SEO functions – keyword competitiveness and ranking factor determination to name two. Below, I discuss a simple, algorithm-agnostic method to determine relative rank gaps and then explain a few key uses of this information.

Exponentially Weighted Moving Average

While search results are ordered 1-10 and have a single integer value each day, over time this position can and will fluctuate as ranking factors are updated and competitors make changes. This fluctuation is represented in the aggregate in systems like SEOMoz MozCast or SerpMetrics Flux. Sometimes we will look at an individual keyword to determine a specific SERP’s volatility but in this case, we will instead look at individual URLs within each keyword SERP over time to determine an average ranking.

As you can imagine, it would be silly to just average the last 30 days of rankings for an individual URL on any given keyword. Doing so would ignore the trajectory of the URL (tending upwards or downwards) and, if that time period included a major update, could have remarkably high variance. Instead, we look to a simple technique used in stock market volatility forecasting called Exponentially Weighted Moving Average (EWMA).

I’ll give a terse explanation here, but please take some time to read up a little more. When you average something over time, you can often assume that the most recent data is more valuable that the oldest data. Yesterday’s rankings are probably a better predictor than your rankings 30 days ago of how your site will perform tomorrow. Instead of averaging all days of rankings data together, we weight them based on recency. The most simple method to accomplish this is a rolling average where each day you find the mean of today’s value and all previous day’s means.

Let’s say you rank #1 today and yesterday you ranked #2. Using EWMA, we would average today and yesterday and come up with 1.5. Another day elapses and you are still at #1. Instead of averaging, #1, #1, and #2, you would simply average today’s rank (#1) with yesterday’s average (#1.5). Your new EMWA is #1.25. In this regard, each new day is worth 50% of the weight in the average, and all previous days are worth a combined 50%.

The magic, of course, is in choosing that weight. Maybe an exponential weight is too strong, and today’s ranking should only matter 1/3. Or maybe we should vary that weight based on a second rolling metric itself, like the degree of change, or a separate weighted variance. (or, most importantly, a weight that preserves rank order) While worth thinking about, let’s go ahead and jump to the good stuff.

Finding Volatility Pockets for Keyword Competitiveness

Let’s say we have now used an EWMA to determine an average rank rather than displayed rank. In the image above, you can see the overlaps that this can create. The first ranking URL might normally rank #1 6 out of 7 days. However, approximately 1 time a week, the #2 spot over takes them. URLs ranking #3 and #4 almost never transpose with one another, and 4 never cedes its position to #5.

However, #5, #6, and #7 quite regularly interchange with one another. And your site currently ranks #8. You haven’t dropped back to 9 in over a month, but your rarely shuffle to #7. What we have done here is used the EWMA as a proxy for competitor’s relative rank score. In this example, because #5, #6, and #7 are all similarly scored, there is a pocket of volatility surrounding them. Because our example site is sitting directly behind this pocket, we have reason to believe that an incremental improvement in value could precipitate a 3 position jump in the SERPs. Of course, we would need to further this analysis with a look at exactly what kind of investment that might entail, but we have a clean, easy to calculate opportunity from which to begin our analysis.

Finding volatility pockets like this that sit around positions #3, #4, and #5 can be particularly valuable because a jump from below the fold to above the fold can yield strong CTR increases from the search results.

Improving Correlation Studies

This one really gets my blood going because of the opportunity it presents for better aggregate studies of rank factors. Some of you might know that a few years back with the help of many including Authority Labs, Majestic SEO, and SEOMoz, we ran a correlation study with over 1,000,000 keywords. This produced many innovations at Virante including one we still use regularly today, relevancy modified MozRank.

One of the biggest issues we run into with the correlation study is that the dependent variable has been tampered with by Google. Instead of giving us the actual rank score, we see the 1 through 10 integer rank. #5 might literally be 100x more relevant to the query than #6, and #6 is only .000000001x better than #7. If we take a snapshot of rankings on just 1 day, we can’t model this out. We are stuck with a fixed, stair-step dependent variable.

But, if we use an exponentially weighted moving average, we can help solve several issues…

1. We can handle the issue of lagging independent variables (like slightly out-dated link data)
2. We can, to some degree, transition rank from discrete to continuous variable
3. We can establish a cohort of solid, unchanging rankings to analyze.

More with Math

The Art vs. Science debate continues on in the SEO community. I don’t think that it is an unhealthy argument, but I do want to make sure that we always stay grounded in math and science. These tell us what matters and in what degree, it helps us determine measurable goals. Art is the method we use to meet those goals.

The integration is actually incredibly useful. In making link prospecting scalable, it is important that you be able to filter prospects based on relevancy. This is the case for nearly any out reach effort, but is particularly difficult in the Broken Link Building strategy because the targets no longer exist. In the past, marketers would need to click through to archive.org to try and find a record of the previous site, or use the URL to guess what the content may have been about. Historically, this process has been difficult to automate. You could use proxies for relevancy like keyword in title or URL neither of which would have been very useful, for example, if you were judging this page for relevancy to the term SEO, despite the fact that its nTopic relevancy is over 98%.

The Broken Link Building tool takes each broken link opportunity, scrapes the Archive.org entry and any anchor text in links pointing to it, and passes those along with the prospecting keywords over to the nTopic API. In the campaign example above, a 404 page was found to the autism-society.org because an SEO blogger had randomly linked to it as part of a charity event. These types of unrelated links show up regularly across the web, but nTopic allows Broken Link Building users to automatically filter them out because their relevancy grades come back quite low (in this case, a grade F). Instead, the user can focus on the MarketLeap 404s, or CivicSEO.com old wordpress seo plugins page, which are more relevant.

This kind of nTopic integraction will continue to spread to search engine optimization services and tools across the web where determining relevancy matters – and in SEO, it matters both on the link building and content creating sides of the coin.