Last Updated on November 30, 2019 by Christopher G Mendla
The deadline is rapidly approaching for Google’s algo change that puts a heavy weight on a site’s mobile readiness.
I have a Joomla 3.4.1 site that, according to Google Mobile Readiness Tool, wasn’t mobile ready. The tool complained about resources being blocked in Robots.txt. The default robots.txt for Joomla blocks the media folder. The tool said it needed access to the media folder.
The list of blocked resources shows the URLs in a truncated fashion but mousing over a URL will show the full address.
The fix was to simply go into .htaccess robots.txt and remark out the line blocking the robots from the media folder (Or simply delete that line).
When I checked Google’s mobile tool again, the site was mobile ready..
Edit note – Many thanks to Randy Garbin for pointing out that I typed .htaccess when the correct file is robots.txt. My apologies for any confusion.
UPDATE – When I checked this month, I found that simply remarking out the lines in robots.txt was no longer working – I had to actually delete the lines.
Updated post can be found here.
Not seeing that rule, Chris. Can you tell me what line it's on?
Any security issues with doing this though?
Apparently new installs of Joomla 3.x do not include the /media folder in robots.txt. However, when you upgrade, the robots.txt is left as-is.
There is a discussion on github about this.
https://github.com/joomla/joomla-cms/issues/5208
=================================================================
COM_CPANEL_MSG_ROBOTS_BODY="A change to the default robots.txt files was made in Joomla! 3.3 to allow google to access templates and media files by default to improve SEO. This change is not applied automatically on upgrades and users are recommended to review the changes in the robots.txt.dist file and implement these change in their own robots.txt file."
================================================================
Most Joomla users, including myself, have very little reason to visit Github.
Pete –
I doubt there are any security issues due to
(1) the nature of robots.txt as a guide , not a firewall
(2) The nature of the files in the /media folder
(3) the fact that Joomla is no longer including /media in the default robots.txt