Hello,
Some sites detect robots and doesn't let WG++ to do its work. For example goalzz.com
I can see the page source clearly in FireFox but WebGrab log shows this:
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script>blah blah blah
This site is using incapsula.com for robots detection.
Is there a workaround solution for this?
Yes, even this info would be perfect...
Do I need to have my own server for scrubbing this info?
Thanks for your help