[This is part of the The Blogger’s Essential Guide to Search Engine Optimization Series.]
Throughout this Blog Series on SEO we’re going to take a look at some of the most important elements of your blog’s information and data, especially the information that the search engines care about.
As I mentioned in the practical introduction to SEO the more you know about search engines the more adequately you’ll be able to leverage and manage them well for your advantage!
As a result I’ve created a new tool that’ll help you visually see a number of the elements that a search engine sees – this way you can start thinking like a Search Engine a tad bit more.
Take a look here: SEO Spider Test & Crawl Tool.
It’s pretty easy to use and once you kick in your blog URL and the anti-bot code then you should see the system spit out something like this:
It might seem like a bit of gibberish but this information is what the Search Engines see when they see your blog and website! Sure, there are some other elements that particular search engines review as well that are related to their particular algorithm but from a pure content-perspective what you see is literally what you get.
Meta Elements: The What and The Why!
Technically the term for these pieces of information meta elements but don’t worry so much about the technical terms – your goal is to be able to both recognize them and understand their importance.
I could stare at this all day!
Meta elements provide the much-needed and helpful information for search engines to categorize them and catalogue their data for future retrieval for your much-needed visitors! This information is directly inserted into the resulting HTML document that is created (via WordPress or another Content Management System or publishing application) as it’s interpreted into the web and browser layer. In other words, you don’t see this information (or you don’t see it the same as a SE does).
The point is that these elements are a huge focus for you and I as it relates to search engine optimization (SEO) and how a SE ranks the sites compared to the many competitors that sit out there ready to gobble up your attention. For a long time up until the mid to late 90′s most SEs relied very heavily on this meta data to not only classify but to rank. Obviously the more you knew or the more you knew how to take advantage of this was a boon to your organization and website as those that knew how to manage and leverage their native meta data on their website saw much higher returns of traffic.
And an entire industry was born: Search Engine Optimizers (that’s what they were called and still are called today), professionals, consultants, and even full-blown businesses rose overnight charging tens of thousands of dollars to help other organizations compete in the growing landscape. Fascinating, right? Some of these organizations did it “right” (we call this White Hat SEO, or legal SEO) and some of these organizations played a bit “dirty” (we call this Black Hat SEO, or illegitimate SEO).
Meta elements today play a slightly different role on the ranking of your site now and their bottom-line utility has wavered a bit – does this mean that you can skip this post and move on from here without caring? No. Not a chance. This is where every blogger that cares about SEO needs to start and covering your bases from the foundation is the most important place to start!
Here’s what you need to ultimately remember: SEO for your blog is about everything that is on it and coming to it. The things on your blog are all the elements you might expect:
- Quantity and quality of content
- Text (spelling and grammar count)
- Images
- Media
- Links (working and broken)
- Meta Data
- Server Information
- Code, Scripts, other 3rd party elements, and technical precision of the work done (pays to get it done right!)
- Speed
- Language
- Uniqueness
- Redundancy
- Geography
- Advertising revenue yield
- Freshness
- And more depending on the specific SE algorithms
And some of the elements that are coming to it are as follows:
- Incoming links
- Volume of searches
- Consistency of returns
- Viewer traffic
- Time within the website directly
- Page views
- Visits and revisits
- Click throughs
- And other elements and factors depending on the specific SE algorithms
We’ll spend much more time in some of these areas in the next few blog posts in this series but this will give you a good and comprehensive overview that there is not only a lot to consider but a lot of opportunity to optimize and take your blog to some serious new heights!
Many of you may never have even considered or even thought about some of these elements – don’t worry! There is a very slim chance that you’re doing anything that’s critically and catastrophically reducing your search engine results but we’ll want to make sure in any case, right?
Meta Elements Represented in the Spider Tool:
So, back to the tool that I’ve released – here’s a breakdown of the elements that are listed via the tool, what they mean, and why they are generally important!
1. URL
This is simply your blog address as understood by the SEs. You’ll notice that it’ll take both http:// and www. or without.
2. Title
This is the actual Title of your site and what the SEs see. It’s important that this relevant to your site content and focus! You might include your “tagline” or catch phrase here or another mission statement-like saying. Relevancy is what is important!
3. Description
This is the description attribute that is support by most major search engines such as Yahoo and Bing. Google will fall back on this tag when information about your blog is requested.
Essentially this element should provide a concise and accurate explanation of what the content of the web page (or your blog) is all about. You have the opportunity here to craft something meaningful, memorable, and contextually relevant to your readers and the SEs, especially if they are unable to come up with it themselves.
Sometimes this information is automatically shown on the search engine returns and queries but sometimes not – the problem is when it doesn’t and it says something completely wrong (or something that changes so often as to confuse the user and the search engine).
Take for example the description of my blog when you Google “TentBlogger”:
Ah, what a great meta description!
My meta description is simple, to the point, and also keyword-rich as it relates to the things that my blog is all about:
Blogging Tips, Tricks, Tools and Practical Teaching covering SEO, WordPress, and Making Money with Your Blog!
And this is what SEO Spider and Crawler Test shows as well:
Not a coincidence! I made it that way!
See how my description is shown here? That’s what the search engine sees because that’s what I’ve asked it to see and that’s what the resulting user gets to see as well.
Simple management and curation of your blog meta information is vitally important, wouldn’t you say?
Finally, one neat thing you might notice is that my meta description has keywords related to my significant pages listed in Google as well. Coincidence? No way – I did that on purpose.
4. Keywords
The keywords attribute has been one of the most popular meta elements that most SEOs have considered to be on the top of the list. However, times have changed (as the algorithms of the SEs have changed) and this is no longer the dominant factor in search engine ranking. This is due to the fact that people tried to “Black Hat” their execution and create spam sites that took advantage of the SE’s systems.
Thus, search engines began to drop the support for this meta information and there is no general consensus as to how important it is in the major search engine systems. In fact, it’s been stated explicitly by some of the more important luminaries of SEO, like Matt Cutts of Google, that Google itself is no longer taking keywords into a strong account in their ranking formula.
Yahoo! and other search engines still do though and so the point of this is that you should still consider it, especially when it’s not too difficult to manage or maintain. So do it!
Here’s what my keywords are as produced by the Spider Tool:
More the merrier?
Here’s what is in in text form:
blog tips, blogging, seo, wordpress, wordpress themes, wordpress plugins, problogger, blog, blogging tips, tools, apps, making money
The natural question is how many keywords you should actually put in there and it varies. I’ve seen that you should keep it light (less than 5 or 6) and then I’ve seen some relatively bogus accounts on doing 30+. I’m pretty comfortable going up to 15 keywords myself (although I currently have just 12).
5. Size
This is the size of the page that’s being selected to test in bytes. You might see a different value in different calculators but this is generally important for one reason and one reason alone: The larger your size the longer it takes to download your information and the longer it takes the longer it takes your readers and search engines to get your content.
Slow performance will impact your ranking negatively! You don’t want a heavy blog and you don’t want to lose ranking and opportunity just because you really really really want that 10mb image and 30mb song to play in the background.
Of course there are tons of ways to reduce the overall page weight and size of your blog, including these strategies:
- Throw out any page elements that aren’t essential.
- Clean up your design and CSS.
- Get rid of any heavy page scripts, frames, or 3rd party elements.
- Compress and optimize your images and media.
- Clean up your code, HTML, and interior elements.
- And much much more…
We’ll cover more of these in a future post in this series very soon but you’ll want to keep these in the back of your mind.
6. Texts
The text area just simply shows all the text that a search engine sees when it indexes and crawls your blog. It should be all your content that you can see directly on your site. Nothing too fancy here.
But, you might want to consider the following meta information that can prove to be quite valuable like the number of words on your homepage (front page of your blog) and distinction.
7. Number of Words
There are a number of competing theories on the perfect number of words on a page that search engines take into consideration in their ranking system. My research and experience has shown that a good page that ranks well has 500 to 3,000 words on it which could be from about 2,000 to 20,000 characters.
Almost 1,500 exactly!
This is a good number to strive for and with a blog there are some distinct ways of managing this to optimize search engine returns, especially like something as simple as the More Tag in WordPress.
As you can see by my results I try to hit right in the middle (1,500 words on the homepage) as often as I can. You can be sure that I do this after I publish a post every single time to make sure I’m in this “sweet spot.”
Heck, it’s really easy for you to do this too now with the Spider and Crawl Tool – every time you hit the publish button come back to the tool and see how many words you’ve got on your homepage. If you’re hitting a bit too high then use that More Tag to reduce the number of words on the main page.
Easy, right? And make sure that your sidebar content isn’t taking up all that valuable text!
8. Number of Distinct Words
The number of distinct words is also important as this is directly related to what we call keyword density. I’m going to spend much more time on this in another blog post but generally this will show you the relative frequency of the word in the text on your homepage.
This is quite easy to calculate and isn’t rocket science by any means: If you have 100 words and you use the same word 5 times then the frequency is 5 or 5% keyword density.
The element that you should be concerned with is having the right mix and frequency – too little and SEs don’t pay attention and too many they will consider it keyword spamming and you’ll be penalized.
As I mentioned before I’ll go into this in much greater depth soon.
9. Keywords
This next section shows you some of the keywords that are currently on your homepage as well as the relative keyword density. We’ll take this information and use it more much later but it will behoove you to take a look and walk through some of your keywords as understood by search engines.
10. Links
This shows you what links the search engines see when they crawl your homepage. These provide valuable information to not only the sites that are directly linked but also in your own blog as well. The better quality links that are relevant and contextual the better you rank.
Also, the number of links that you have play a role as well. It has well been understood that you should have no more than 100 links per indexable page. As you can see the number of links that I have is currently 64, well within that region. You should optimally shoot for this as well.
So yummy.
11. HTML Source Code
Finally, if you should ever need to see the actual HTML code of your blog and site it’s provided there as well. We’ll dig into this section more as well but this is what the search engine interprets when it’s crawling and spidering your site.
Well that’s that! There is a general overview of not only the tool but the resulting meta information that you’re going to want to care about.
Let me know if you have any questions and please feel free to share the new Spider and Crawl Tool with others! Thanks so much guys!
By the way, I love building apps for you all – makes my heart smile.
[This is part of the The Blogger’s Essential Guide to Search Engine Optimization Series.]
This is full of #win
re-blog. ftw!
to many words
i think you meant “too many words”.
#ROFL!
Love you man.
How do you set the description and keywords on individual posts? I don’t want the same description or keywords showing site-wide. Each post touches on different topics. Currently I’m using Yoast’s SEO plugin but I know you’ve said in the past you don’t see the need for the plugins but I haven’t figured out a way to “automate” these differences from post to post.
yoast is a great tool to use and i’ll have a review of that in this series. it’s a good place to start!
I use Yoast too.
I noticed that this spider tool doesn’t recognize the meta data put in by Yoast WordPress SEO plugin.
At first, I thought maybe it was how I had the plugin set up. So I found several other blogs which use Yoast, and they all had the same problem. Even Yoast.com.
So then I thought maybe it was the Tentblogger spider tool. So I found a few other spider tools, and experienced the same problem with them all.
So it must be the plugin itself. Are spiders really not reading the meta data from the plugin?
it should pull your information (the tool that is).
Right, but it doesn’t. Nor does the tool pull it for other sites that use the plugin. Other spider test tools around the web also do not pull the information, so it’s not just yours.
e.g.
http://www.searchenginegenie.com/spider-test-tool/index.php
http://tools.seobook.com/general/spider-test/
Here are some sites I tried on all of these tools:
http://yoast.com/
http://vintage-ring.com/
jeremy,
i see it working… even with yoasts.
http://cl.ly/7toW
Hmmm. I wish I had taken screenshots from before….
I just went and checked, and the other spider tools I listed above still do not show the meta data, which means you have the best spider tool on the web!
http://www.flickr.com/photos/64508176@N08/5866538786/
http://www.flickr.com/photos/64508176@N08/5866535020/
Since yours now sees it and the other spider tools still don’t, I wish I knew whether or not the search engine spider tools were seeing the yoast meta data or not…
hah! i updated mine to include alternate forms of the code… for example, some skip if you use single or double quotation marks. most of those tools need to be updated… but i try to keep mine up to snuff.
Ah ha! Awesome.
You truly have the best spider tool on the web!
I am trusting that your tool matches the spider tools that the search engines use?
no, i just made it up for fun and games.
……….
……….
……….
comon jeremy…..!
It wasn’t a dumb question. There are numerous spider tools out there which give different results.
How do we know which type of spider the search engines are using?
I HOPE their tools are like yours (now that you have updated it), but is it possible that their tools are like one of those other tools?
As it turns out, it appears your spider tool is different than Google’s after all, so my question was quite valid.
I have not changed my site title or description in a few months, and here is what Google has as my Description:
http://www.flickr.com/photos/64508176@N08/5879100779/
The description doesn’t actually show up… Google shows no description.
But here is what shows up on your spider tool:
http://www.flickr.com/photos/64508176@N08/5879662500/
So the two spiders are pulling different information.
jeremy, bro, chill out. why in the world would i ever suggest that my crawler tool is the same as google’s so-called “tool”? mine was built in 4 hours, google’s was built by a team of world-class engineers and billions of billions of dollars…
people create these tools all the time to help assess what the search algorithms are but no one knows exactly. sheesh bro!
the point of the tool is to help people understand the basic mechanics of search engines and SERPs, not to replicate (or even attempt to replicate) the billion-dollar machine that is google (or bing, yahoo, etc). that’s why the tool sits in an “intro to meta elements” blog post…
not sure where you’re trying to go with this…?
also, i see some other meta desc: http://cl.ly/7zVq
John,
I think we have a bit of a misunderstanding. Blog comments don’t relay tone of voice. I’m not upset.
I just wanted to make sure search engines were seeing my meta tags.
Here is a summary of our exchange:
Me: Cool tool. But it’s not showing my meta data.
You: Now it should. I made some changes to it.
Me: Yep, it does now. Awesome. So does your spider tool matches those from search engines?
You: Of course! Why do you think I made it?
Me: But the results are way different. Your (updated) spider tool is now reading my meta tags. Google still isn’t.
You: That’s because my spider tool is different than their spider tool.
…And that is where we are now…
… sure. understandable. my previous comments had hints of sarcasm. you didn’t pick up on those.
Tools like this are very useful if you know some what a little about seo. But you have to use them for them to work.
“Black Seo Guy “Signing Off”
thank blackseoguy… i’m pretty sure that’s what i mentioned in the post.
I don’t think he reads the posts anywhere. Just comments. Seriously.
… #ROFL . you just made my day.
OMG, what a tool.. Pun intended
Thanks for sharing the tool. Did the test. Good to see what the SEO see. Keep pushing!
sure thing dewitt!
Hi, great article!
I’m lost on something, we just launched our church online service in spanish, and we’re trying to get into google, so we filled our title and keywords, but when I use your spider test I get empty results
and maybe google gets empty results too because the description it’s not listed on the results page.
The domain is: http://www.laiglesia.tv
Can you guys give any hint on what’s wrong?
Thanks a lot!
it seems that the language is in EN not spanish? “ES” .
Thank you John for taking some time and answering my question, churchonline.org does not offer (for now) their plattform in Spanish, so we’re translating many of the elements with jQuery after the page loads. I think that’s going to be a problem for our seo goals.
Looking forward to the rest of the series!
Thanks!
ah. ok! sure thing esteban!
WOW! This series is already getting awesome. The Spider Test is a awesome tool that helped me immediately identify some issues that I need to address with SEO. Thanks for making it available.
sure thing JD!
Wow, this is awesome! Are you going to go into how we can fill in things like the description and keywords?
yes, for sure!
Hi John,
Thanks for the excellent post. Reading and implementing what you are teaching. I’ve checked out my site in your tool, and I’ve not got any keywords, and a description I don’t like.
Where do I change these? Is it in the theme or the wordpress settings? I’m running the awesome standard theme 2.5.
Thanks
Rob – I believe you can set your Description in WP under Settings|General|Tagline.
Jon – I have the same question as Rob in that I’m not sure where I add Keywords. I’m not seeing where I go to put those in and thus my Spider View shows no Keywords.
andy,
we are definitely going to talk about these…. don’t worry!
rob,
we’ll cover this. you could copy and paste them in manually, but there will be some other options. don’t worry!
Okay, so when you mentioned the links you said you had 64, did you go through and manually count them or am I just missing where that number is displayed.
i manually counted them.
copy and paste them into excel and you can see much quicker than counting.
yeah, that’s what i did. i should have mentioned that.
Thanks for this John. I have been following you the last couple of months, since I have to make a concerted effort to get my blog traffic up, in order to monetize my blog properly.
Looking forward to your following post on how to get the Descriptions and keywords correct, as well as all the other stuff that I don’t know about yet.
thanks for dropping a comment! appreciate the support!
Thanks John!! Looking forward to the next one
So I counted and have about 150 links. I am assuming that I should move my purchase links to the bottom of the page so the spider is only seeing them on the particular pages and not on the home pages. I have a lot of posts links going down the side. How many do you think is too many for SEO?
honestly you should try to stay under 100 per page. repetition isn’t good as it can be seen as spammy.
I reorganized and dropped down to 87 before moving the purchase links. That would move around 30 off the man page. Now to cut word (around 2500) and add keywords.
By the way since switching to bookwi.se I have more than trippled my rss subscribers and more than doubled my page views in three months.
now that is dope. increase of sales too?
i knew that bookwi.se would be awesome!
Not really. I get more sales when Amazon has a sale. Early this month there was a big sale on kindle books, had 70 books sold. That was more income (although 6% of $1.99 isn’t much) than previous 3 months combined. Year to date slightly higher than last year. Just seems to happen more by chance than design. But it should be enough to cover hosting and some give aways.
starts with traffic…!
i’m pretty stoked!
@jeremy,
i can’t vouch for their tools but there’s enough difference to have more than a few tools to use.
Wow, this is so awesome. I’ve been blogging for a while now, just typing away, and now I’m trying to wrap my head around the back end stuff. This is a great start and such a great series! Thanks so much for putting in the time and putting it out there for us to use!
thanks for stopping by libby!
Wow, that tool was awesome! One thing I saw was that my word count is way too high (well, still below 3,000 but barely). The cause is my sidebar which has lots of text. Need to find a solution for that I guess. Thanks for sharing, very helpful!
sure thing rachel!
Awesome tool! Just got around to trying it.
So if my word count is too low (854) should I increase it by having more posts displayed on the main page? I think mine is 5 now. Or show more of the content before the “continue reading” link?
Also how do you get those kw under the meta description when your page comes up in Google?
you’ll want to add them manually if you can in your theme file (header.php).
remember, there’s no “law” that says you need more word count on the front, just patterns of success.
Thanks John. Bought the Standard Theme last night and I’m working on moving my site to the Standard theme. Have it installed locally (thanks to you) and am working on customizing it.
Just missed seeing that coupon code by mins but I emailed the 8bit team. Waiting to hear back. Thanks!
sweet! we’ll get to it!
Hi John,
Do you explain anywhere how it is that you manipulate your site’s meta description (as opposed to on the posts themselves)? My apologies if it’s been hiding in plain sight; I just haven’t been able to locate it in your SEO writings.
Thanks!
Scott
i believe I have mentioned this in a previous post. you have to change it in the header file.
Hello!
I’m enjoying the wealth of knowledge you sharing on your blog. Kudos to you!
A couple questions.
Question 1.
~~~~~~~~~~~~~~~~
Item #10, Links – No more than 100 links per indexable page is the recommendation. Does that work against an “Archive” page that has several hundreds of links (a unique link to every blog post) or are you referring to set of links that show up consistently on every page (ads, sponsors, blog rolls, link list of favorite websites / products / cams / online games, etc.)?
Question 2.
~~~~~~~~~~~~~~~~
Keywords – should they be only for the home page or should they be in the head section of HTML on every page with keywords that are relevant to the page they are attached to? I’ve looked through a few of your pages and found no meta content keywords except for on your home page.
I still have a ton to read yet on your blog. Thanks for so much good info!
Take care!
Jon
the rule isn’t hard and fast so you can break it every so often… in my archives page it’s obvious that i break that rule…! but it’s helpful for users and google can respect that!
keywords can be used in every single page… but note that many search engines, like google, don’t really care much for them. meta descriptions matter a bit more.
http://www.stakepower.com is a website value calculator, estimations and information tool. It is a free
website where you can calculate your website value, number of daily visitors, website worth,
website information and many other information
thanks … stakepower..
In Safari or Firefox, when I try to use the Spider Test nothing happens. I click the “view” button and text briefly appears below it, but nothing happens.
Please help!
I’m seeing the same thing! Would love to check this tool out John!
cool. this needs some updating… i’m going to try to get to this soon.
Fixed it. give it a go.
Awesome. Thank you!
Now I just have to interpret my findings. The tool shows that I don’t have a title for my blog, when in WP in Standard theme I clearly have a title in my General Setting.
Harumph.
Same problem. John where should we be editing the title and keywords in Standard Theme
in the header.php file. i’m going to submit a change to standard to improve this.
Thanks
Hi John,
I just came across your site few days before and I must say that I am spending a lot of time reading your articles & trying to put things into practice.
I have a blog about Microsoft Exchange Server and has good content (I hope), http://www.howexchangeworks.com. But, when I check the site with your tool, I only have 119 keywords and 74 distinct words. What am I doing wrong? I was on blogger till last week and just switched over to wordpress self hosting.
Any help is much appreciated.
Thanks.
something in your theme is cutting things off.
John,
Thanks for this article. This is awesome!
Matt
Excellent post here with useful information except for the malfunction SEO Spider Test & Crawl Tool. I could only see the html code at the location that a search box should appear. Kindly help to fix it.
Hi
Awesome content and explained in a very lucid way :clap: :clap:
The spider tool is broken
Spider Tool is broken!!
i’ll have to take a look.
Hi. Has the SEO Spider Test & Crawl Tool been taken down? Every time I click on the link, I am taken to your most recent blog post.
yeah, it’s been removed.