RoboGen is a visual editor for Robot Exclusion Files.
It allows the user to quickly and easily create the ROBOTS.TXT files required to instruct web search engines which parts of a web site are not to be indexed and made searchable by the general web public.
RoboGen does this by providing the user a way to log onto his FTP server and then select the documents and directories which are not to be made searchable.
RoboGen maintains a database of over 180 search engine user-agents, which are selectable from a drop down menu.
Это небольшая программа, которая призвана помочь вам в поиске так называемого дублированного контента. А проще говоря копий какого-то текста в Интернет.