Results 1 to 17 of 17

Thread: Which Bots crawl your mobile websites?

  1. #1
    Mobility Regular
    Join Date
    Jun 2008
    Posts
    306

    Default Which Bots crawl your mobile websites?

    I was just wondering which Bots crawl your mobile websites?

  2. #2
    Senior Member andymoore's Avatar
    Join Date
    Aug 2008
    Posts
    1,612

    Default

    http://pml.mobi/ monitor.php?where=all

    My realtime monitor showing all page views on my site.

    Link broken as I don't want it indexing

  3. #3
    Mobility Regular
    Join Date
    Jun 2008
    Posts
    306

    Default

    Thanks for the link! Wow, there is a lot of activity on your website.
    Do you have some advice for promoting a mobile website and how do
    you get those reports?

  4. #4
    Senior Member andymoore's Avatar
    Join Date
    Aug 2008
    Posts
    1,612

    Default

    No worries.

    Yes, a lot of activity on the site, the spiders are going mad with the new layout, new content, new everything...

    Google has been the busiest, they've crawled over five million pages since the start of June. That is just with user agents that mention Google, how many pages they view as a regular browser or mobile I don't know.

    Google bot has viewed a 100 pages in the last 2 minutes, 33 of those were viewed as a mobile device.

    Promoting a mobile site, well I reckon a lot of it is like promoting a regular site, first off you need great content that people will want to share and link to, second you need a lot of links from the most relevant sites you can find. Third and most important the desire to never give in.

    Most if not all the SEO I do is on page, I hate emailing people asking for links so very rarely do it, I've always been a fan of valid markup, add to that a sound / solid navigational structure and ensure that your site is crawlable, or deep crawlable for those with lots of pages. Keywords in: anchor text in links, page titles, h1 tags, h2,h3, p, b, i and so on. Copy written for humans for not search engines, fast and stable hosting as you can....... I'll finish my SEO checklist site some time.

    As for the report, my site is built in PHP and my title tags are all a value, I log every page view in a MySQL database with the title, URL viewed, referring URL, HTTP status, IP address, mobile or not, date and how long it took the script to complete.

    Then the report itself is just one small script, part one of the script shows the HTML with an AJAX call to read the second part of the script and update the page contents every second.

    The bit in the report that mentions the actual position where we ranked in the SERPS is done by checking for Google URLS that go like this google.(something)/url? - it then breaks down the referring URL to extract the cd position which contains the ranking.

    http://pml.mobi/ monitor.php?where=pos

    Getting the search terms used isn't too hard either when they're usually in the referring URL as a q= value.

    As for the mobile or not values in the report, no prizes for guessing how I worked that bit out...

    I have had thoughts about making a version of this script that's easy to install and giving it away so others can learn from the data it produces.

    If I were to set it up so I could give that code away I'd keep it simple and just make it log the following:

    Page title
    URL viewed
    Referring URL
    Mobile
    Date

    Is that something, had I the time to make it, that folks would want for free?

    I'll warn in advance though, the database can get pretty big......
    Andy Moore
    i don't do tech / mobi anymore - why do something that annoys and stresses lol

    New: Most Minty Tooth Oil - GMO and fluoride free toothpaste alternative

    100% natural botanical extracts to whiten teeth, freshen breath and strengthen gums
    Used, recommended and sold by Harley Street private dentists

    My other websites:

    http://sellyoursongs.info/ - Sell Your Songs using Mobile Payments
    http://detectmobilebrowsers.mobi/ - Detect Mobile Browsers
    http://andymoore.info/m-commerce-php/ - M-Commerce PHP (sell stuff on the mobile web)
    http://locationdetection.mobi/ - Detect a mobile browser's latitude & longitude
    http://andymoore.info/mysql-lyrics-database/ - Half a million lyrics


  5. #5
    Senior Member ChinaMobi's Avatar
    Join Date
    May 2010
    Posts
    1,593

    Default

    Excellent explanation. Thanks Andy for this valuable information. This also shows that there's a lot I need to learn.

    Quote Originally Posted by andymoore View Post
    The bit in the report that mentions the actual position where we ranked in the SERPS is done by checking for Google URLS that go like this google.(something)/url?
    Don't quite understand the Google part. Can you give an example? How did you get the SERPS information off Google? Content scraping or API call using Ajax?

    Once again, I appreciate your willingness to share your work with us.

  6. #6
    Senior Member andymoore's Avatar
    Join Date
    Aug 2008
    Posts
    1,612

    Default

    No worries!

    No scraping, no api, just using data in the URL that sent the visitor my way.

    If page a links to page b the referrer is page a, if the URL of page a contains /url?......cd=XX...... you can grab the value of cd to get the position value.

    This happens when google doesn't link direct to you but instead links to a google URL that redirects you.

    Call me a geek but I love data like user agents and referrering addresses....


  7. #7
    Mobility Regular pilot's Avatar
    Join Date
    Jun 2007
    Posts
    812

    Default

    Andy,

    Your realtime monitoring system rocks! Nice job!

  8. #8
    Senior Member andymoore's Avatar
    Join Date
    Aug 2008
    Posts
    1,612

    Default

    I've dragged myself off the bean bag outside and back to my desk so I give some better examples:

    http://www.google.com/ url?sa=t&source=web&cd=7&ved=0CDEQFjAG&url=http%3A %2F%2Fweb2txt.co.uk%2Flady-gaga-fashion-ringtone-353521%2F&rct=j&q=lady%20gaga%20fashion%20ringtone &ei=zxVgTMmWJ5WJONTnsZgP&usg=AFQjCNH-vDFthZWFYU_Mwp4PxO2Sum3fww&sig2=PFJAwcZk6jEj69vimH TEQw

    If clicked (clickable) that link, which was a referrer a few moments ago, would redirect you through to my site.

    If we take it to bits:

    http://www.google.com/
    url? = it's a redirect with the juicy info we want
    sa=t& = no idea
    source=web& = from the web
    cd=7& = THE JUICY BIT
    ved=0CDEQFjAG& = pass
    url=http%3A%2F%2Fweb2txt.co.uk%2Flady-gaga-fashion-ringtone-353521%2F& = where the visitor was redirected to
    rct=j& = pass
    q=lady%20gaga%20fashion%20ringtone& = the actually query the user entered

    ei=zxVgTMmWJ5WJONTnsZgP& = pass
    usg=AFQjCNH-vDFthZWFYU_Mwp4PxO2Sum3fww&sig2=PFJAwcZk6jEj69vimH TEQw = pass

    As you can see there are two juicy bits, cd= and q=, q= is always present and cd= is only there when there's also a /url? in the referrer.

    I know SEO firms that don't know about this!

    PHP to do it....

    PHP Code:
    $position explode('&cd=',$referrer);
    $position explode('&',$position['1']);
    $position =    $position['0']; 
    @pilot Thanks!

    Andy Moore
    i don't do tech / mobi anymore - why do something that annoys and stresses lol

    New: Most Minty Tooth Oil - GMO and fluoride free toothpaste alternative

    100% natural botanical extracts to whiten teeth, freshen breath and strengthen gums
    Used, recommended and sold by Harley Street private dentists

    My other websites:

    http://sellyoursongs.info/ - Sell Your Songs using Mobile Payments
    http://detectmobilebrowsers.mobi/ - Detect Mobile Browsers
    http://andymoore.info/m-commerce-php/ - M-Commerce PHP (sell stuff on the mobile web)
    http://locationdetection.mobi/ - Detect a mobile browser's latitude & longitude
    http://andymoore.info/mysql-lyrics-database/ - Half a million lyrics


  9. #9
    Senior Member ChinaMobi's Avatar
    Join Date
    May 2010
    Posts
    1,593

    Default

    Thanks Andy, I've learned something valuable. So when Google sends someone to your site, it also gives you info about the user.

  10. #10
    Mobility Regular morency's Avatar
    Join Date
    Nov 2008
    Location
    23.8134710 , 44.3247600
    Posts
    213

    Default

    2 hot andy ...you should write some blog post about SEO ... I see a lot of traffic coming from google :-O

  11. #11
    Mobility Regular
    Join Date
    Jun 2008
    Posts
    306

    Default

    Thanks Andy for the detailed information. Maybe you can help me with some questions I have?

    -How do you run a PHP script when a link is clicked?

    -Is there a way to automatically import the access log (/log/access_log) to my MySQL DB so I can use PHP scripts to filter the information I want?

    -Iím using Webalizer, but I can only see the top 30 referrers. Do you know what identifies a referrer? I have looked at a few in the access log, but havenít found anything they have in common.

  12. #12

  13. #13
    Senior Member andymoore's Avatar
    Join Date
    Aug 2008
    Posts
    1,612

    Default

    Thanks folks.

    Quote Originally Posted by freeflow View Post
    How do you run a PHP script when a link is clicked?

    Is there a way to automatically import the access log (/log/access_log) to my MySQL DB so I can use PHP scripts to filter the information I want?

    Iím using Webalizer, but I can only see the top 30 referrers. Do you know what identifies a referrer? I have looked at a few in the access log, but havenít found anything they have in common.
    1) My entire site is built with PHP so every URL and every click goes through PHP, I use htaccess to rewrite the URLs so they are keyword rich, human and search engine friendly.

    Say you had a HTML site and can run PHP and htaccess (but don't), it is possible to get htaccess to run a script before or after the page has done it's thing.

    php_value auto_prepend_file /full/path/to/log.php

    Then log.php could do it's thing like connect to the database and insert that pageview before the html file is served. I found out about that trick some time back thanks to Dave Child's awesome site: http://www.addedbytes.com/articles/c...output-in-php/

    This for example can help you set up URL rewriting: http://www.addedbytes.com/cheat-shee...e-cheat-sheet/ - the cheat sheets on his site are well worth printing off, laminating and sticking on the wall.

    2) Not sure, I know there are log file tools out there but I don't use them. I just use conventional analytics and log each page view locally too.

    3) The referrer is a HTTP header that is sent along with requests that have followed a link being clicked.

    This explains it better than I could: http://en.wikipedia.org/wiki/HTTP_referrer

    If you were logging all the page views on your site you could then run a report that groups the results by volume, in other words, your own custom made most popular referrers report.

    Needing much more coffee before I feel like I can really function this morning.....
    Andy Moore
    i don't do tech / mobi anymore - why do something that annoys and stresses lol

    New: Most Minty Tooth Oil - GMO and fluoride free toothpaste alternative

    100% natural botanical extracts to whiten teeth, freshen breath and strengthen gums
    Used, recommended and sold by Harley Street private dentists

    My other websites:

    http://sellyoursongs.info/ - Sell Your Songs using Mobile Payments
    http://detectmobilebrowsers.mobi/ - Detect Mobile Browsers
    http://andymoore.info/m-commerce-php/ - M-Commerce PHP (sell stuff on the mobile web)
    http://locationdetection.mobi/ - Detect a mobile browser's latitude & longitude
    http://andymoore.info/mysql-lyrics-database/ - Half a million lyrics


  14. #14
    New Member
    Join Date
    Jun 2012
    Posts
    3

    Default Re

    Thanks andy for sharing the information.

  15. #15
    New Member
    Join Date
    Jun 2012
    Posts
    4

    Default

    Excellent information. This has good information. Thanks for sharing.

    Regards,
    SibzSolutions.com

  16. #16
    New Member
    Join Date
    Feb 2013
    Posts
    3

    Default

    Thanks andy for your valuable info

  17. #17
    New Member
    Join Date
    May 2012
    Posts
    5

    Default Mobile SEO

    I am thinking to start work on mobile SEO and i was trying to get some information about it thanks for sharing tips about mobile SEO.

Similar Threads

  1. Google vs Yahoo vs Bing ~ Crawl vs Clicks
    By andymoore in forum Mobile Search
    Replies: 7
    Last Post: 03-09-2011, 11:09 AM
  2. Replies: 15
    Last Post: 12-08-2010, 06:44 AM
  3. +++ Looking for Mobile Websites +++
    By MobileDesigner in forum News & Discussion
    Replies: 11
    Last Post: 09-23-2009, 12:10 PM
  4. Free mobile websites
    By Binaryman in forum Mobile Advertising
    Replies: 3
    Last Post: 01-04-2009, 05:50 PM
  5. mobile websites
    By svfox in forum News & Discussion
    Replies: 1
    Last Post: 05-13-2008, 11:40 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
A Yup.mobi Mobile Site
Mobile Dating