You are here

ROBOTs detecting sites

4 posts / 0 new

Log in or register to post comments

Last post

Wed, 2017-01-18 23:36

#1

rapidiptv

Offline

rapidiptv's picture

Joined: 8 years

Last seen: 4 months

ROBOTs detecting sites

Hello,

Some sites detect robots and doesn't let WG++ to do its work. For example goalzz.com

I can see the page source clearly in FireFox but WebGrab log shows this:

<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script>blah blah blah

This site is using incapsula.com for robots detection.

Is there a workaround solution for this?

Thu, 2017-01-19 12:44

#2

rapidiptv

Offline

rapidiptv's picture

Joined: 8 years

Last seen: 4 months

Yes, even this info would be perfect...

Thu, 2017-01-19 12:52

#3

rapidiptv

Offline

rapidiptv's picture

Joined: 8 years

Last seen: 4 months

Do I need to have my own server for scrubbing this info?

Thu, 2017-01-19 13:12

#4

rapidiptv

Offline

rapidiptv's picture

Joined: 8 years

Last seen: 4 months

Thanks for your help

Log in or register to post comments

Brought to you by Jan van Straaten

Program Development - Jan van Straaten ------- Web design - Francis De Paemeleere
Supported by: servercare.nl