Robots, bots, crawlers or spiders work for search engines and crawl the web gathering information on new web pages, updating old pages and deleting dead pages. They follow links, some can run javascript and most of them hate Flash.
Digital Agency for Third Sector
