User-agent: * Disallow: /media/* User-agent: Twitterbot Disallow: