DueDil Crawler

DueDil is not interested in using content from websites whose authors do not consent. To remove your data from our website, place a robots.txt file at the top level of your site (e.g. www.yourdomain.com/robots.txt).

The robots.txt file will do two things:

  • It will remove documents from your domain from DueDil.
  • It will tell us not to crawl your site in the future.

To exclude DueDil’s crawler while allowing all other robots to crawl your site, your robots.txt file should say:

User-agent: ElectricMonk
Disallow: /

Robots.txt can be used to block access to the whole domain, or any file or directory within. There are a large number of resources for webmasters and site owners describing this method and how to use it, including http://www.robotstxt.org/.

If you cannot put a robots.txt file up, please get in touch with us at info@duedil.com.