#8191 – Site anaylsis and sitemap issues

Hi Yannick,

The site analysis has finally completed and the dashboard says it has created a sitemap, but the site map does not exist on the server. Also, the number of pending pages are continuously counting up (but randomly timed) . I have checked the DB table "collected_urls" and these are good urls, so I don't really get why no sitemap has been created and why are more URLs are being constantly found, when the site analysis has just run?

Sorry we cannot give admin rights, but can offer a remote session to client that has access if you need to check things.

Thanks for your support

Yes manually started from the admin side.

How does that work, the url points to a file and when browsed it's not found: https://www.xxxx.com/sitemap-4seo.xml

Yes they are default, the Collect incoming requests, is set to "No" and there are no Exclude URLs/Include URLs have been added.

If you want we can allow you to remote on to a PC which has administrative access to the website for a monitored session.

Thanks

The server returns xml for other sitemaps, so I don’t think it is misconfigured, please see here: https://www.xxxx.com/sitemap_xml.xml

We don't use AdminTools.

Yes, its a physical file. The htaccess is standard, as far as we can see. Do you know what should we be looking for regarding configuration please?

Thank you, but no such luck, the only mention of xml in the htaccess file are standard deflate and expire headers...

ExpiresByType text/xml "access plus 0 seconds"
AddOutputFilterByType DEFLATE application/xml

Thanks, please find below...

RewriteOptions inherit
<IfModule mod_headers.c>
  <FilesMatch "\.(ttf|ttc|otf|eot|woff|woff2|font.css|css|js)$">
    Header set Access-Control-Allow-Origin "*"
  </FilesMatch>
</IfModule>

# gtranslate config
RewriteRule ^(en|bg|hr|cs|da|nl|fi|fr|ka|de|el|hu|ga|it|ja|no|fa|pt|ru|es|sv|cy|pl)/(en|bg|hr|cs|da|nl|fi|fr|ka|de|el|hu|ga|it|ja|no|fa|pt|ru|es|sv|cy|pl)/(.*)$ /$1/$3 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(en|bg|hr|cs|da|nl|fi|fr|ka|de|el|hu|ga|it|ja|no|fa|pt|ru|es|sv|cy|pl)/(.*)$ /gtranslate/gtranslate.php?glang=$1&gurl=$2 [L,QSA]
RewriteRule ^(en|bg|hr|cs|da|nl|fi|fr|ka|de|el|hu|ga|it|ja|no|fa|pt|ru|es|sv|cy|pl)$ /$1/ [R=301,L]

##
# @package    Joomla
# @copyright  Copyright (C) 2005 - 2014 Open Source Matters. All rights reserved.
# @license    GNU General Public License version 2 or later; see LICENSE.txt
##

##
# READ THIS COMPLETELY IF YOU CHOOSE TO USE THIS FILE!
#
# The line just below this section: 'Options +FollowSymLinks' may cause problems
# with some server configurations.  It is required for use of mod_rewrite, but may already
# be set by your server administrator in a way that dissallows changing it in
# your .htaccess file.  If using it causes your server to error out, comment it out (add # to
# beginning of line), reload your site in your browser and test your sef url's.  If they work,
# it has been set by your server administrator and you do not need it set here.
##

## Can be commented out if causes errors, see notes above.
Options +FollowSymLinks

## Added in Joomla 13.9.13
<IfModule mod_headers.c>
Header always set X-Content-Type-Options "nosniff"
</IfModule>

## Mod_rewrite in use.

RewriteEngine On

## Begin - Rewrite rules to block out some common exploits.
# If you experience problems on your site block out the operations listed below
# This attempts to block the most common type of exploit `attempts` to Joomla!
#
# Block out any script trying to base64_encode data within the URL.
# Block out any script that includes a <script> tag in URL.
# Block out any script trying to set a PHP GLOBALS variable via URL.
# Block out any script trying to modify a _REQUEST variable via URL.
# Return 403 Forbidden header and show the content of the root homepage
RewriteCond %{QUERY_STRING} base64_encode[^(]*\([^)]*\) [OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
RewriteRule .* index.php [F]
#
## End - Rewrite rules to block out some common exploits.

## Begin - Custom redirects
#
# If you need to redirect some pages, or set a canonical non-www to
# www redirect (or vice versa), place that code here. Ensure those
# redirects use the correct RewriteRule syntax and the [R=301,L] flags.
#
## End - Custom redirects

##
# Uncomment following line if your webserver's URL
# is not directly related to physical file paths.
# Update Your Joomla! Directory (just / for root).
##

# RewriteBase /

## Begin - Joomla! core SEF Section.
#
RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
#
# If the requested path and file is not /index.php and the request
# has not already been internally rewritten to the index.php script
# and the request is for something within the component folder,
# or for the site root, or for an extensionless URL, or the
# requested URL ends with one of the listed extensions
# and the requested path and file doesn't directly match a physical file
# and the requested path and file doesn't directly match a physical folder
# internally rewrite the request to the index.php script
RewriteCond %{REQUEST_URI} !^/index\.php
RewriteCond %{REQUEST_URI} /component/|(/[^.]*|\.(php|html?|feed|pdf|vcf|raw))$ [NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !^/[0-9]+\..+\.cpaneldcv$
RewriteCond %{REQUEST_URI} !^/\.well-known/pki-validation/[A-F0-9]{32}\.txt(?:\ Comodo\ DCV)?$
RewriteRule .* index.php [L]
#
## End - Joomla! core SEF Section.

#Error Handling Pages Custom
ErrorDocument 403 /403

#<Files 403.shtml>
order allow,deny
allow from all
#</Files>


# BEGIN cPanel-generated php ini directives, do not edit
# Manual editing of this file may result in unexpected behavior.
# To make changes to this file, use the cPanel MultiPHP INI Editor (Home >> Software >> MultiPHP INI Editor)
# For more information, read our documentation (https://go.cpanel.net/EA4ModifyINI)
<IfModule php7_module>
   php_flag display_errors Off
   php_value max_execution_time 180
   php_value max_input_time 300
   php_value max_input_vars 1000
   php_value memory_limit 6024M
   php_value session.gc_probability 90
   php_value session.gc_maxlifetime 1400
   php_value session.save_path "/var/cpanel/php/sessions/ea-php56"
   php_value upload_max_filesize 128M
   php_value post_max_size 1024M
   php_flag zlib.output_compression On
</IfModule>
<IfModule lsapi_module>
   php_flag display_errors Off
   php_value max_execution_time 180
   php_value max_input_time 300
   php_value max_input_vars 1000
   php_value memory_limit 6024M
   php_value session.gc_probability 90
   php_value session.gc_maxlifetime 1400
   php_value session.save_path "/var/cpanel/php/sessions/ea-php56"
   php_value upload_max_filesize 128M
   php_value post_max_size 1024M
   php_flag zlib.output_compression On
</IfModule>
# END cPanel-generated php ini directives, do not edit

## START JSPEED OPTIMIZATIONS ##
<IfModule mod_expires.c>
  ExpiresActive on

# Default
  ExpiresDefault "access plus 1 year"

# Application Cache
  ExpiresByType text/cache-manifest "access plus 0 seconds"

# HTML Document
  ExpiresByType text/html "access plus 0 seconds"

# Data documents
  ExpiresByType text/xml "access plus 0 seconds"
  ExpiresByType application/xml "access plus 0 seconds"
  ExpiresByType application/json "access plus 0 seconds"

# Feed XML
  ExpiresByType application/rss+xml "access plus 1 hour"
  ExpiresByType application/atom+xml "access plus 1 hour"

# Favicon
  ExpiresByType image/x-icon "access plus 1 week"

# Media: images, video, audio
  ExpiresByType image/gif "access plus 1 year"
  ExpiresByType image/png "access plus 1 year"
  ExpiresByType image/jpg "access plus 1 year"
  ExpiresByType image/jpeg "access plus 1 year"
  ExpiresByType image/webp "access plus 1 year"
  ExpiresByType video/ogg "access plus 1 year"
  ExpiresByType audio/ogg "access plus 1 year"
  ExpiresByType video/mp4 "access plus 1 year"
  ExpiresByType video/webm "access plus 1 year"

# X-Component files
  ExpiresByType text/x-component "access plus 1 year"

# Fonts
  ExpiresByType application/font-ttf "access plus 1 year"
  ExpiresByType font/opentype "access plus 1 year"
  ExpiresByType application/font-woff "access plus 1 year"
  ExpiresByType application/font-woff2 "access plus 1 year"
  ExpiresByType image/svg+xml "access plus 1 year"
  ExpiresByType application/vnd.ms-fontobject "access plus 1 year"

# CSS and JavaScript
  ExpiresByType text/css "access plus 1 year"
  ExpiresByType text/javascript "access plus 1 year"
  ExpiresByType application/javascript "access plus 1 year"

  <IfModule mod_headers.c>
    Header append Cache-Control "public"
    <FilesMatch ".(js|css|xml|gz|html)$">
       Header append Vary: Accept-Encoding
    </FilesMatch>
  </IfModule>

</IfModule>

<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/html
AddOutputFilterByType DEFLATE text/css
AddOutputFilterByType DEFLATE text/javascript
AddOutputFilterByType DEFLATE text/xml
AddOutputFilterByType DEFLATE text/plain
AddOutputFilterByType DEFLATE image/x-icon
AddOutputFilterByType DEFLATE image/svg+xml
AddOutputFilterByType DEFLATE application/rss+xml
AddOutputFilterByType DEFLATE application/javascript
AddOutputFilterByType DEFLATE application/x-javascript
AddOutputFilterByType DEFLATE application/xml
AddOutputFilterByType DEFLATE application/xhtml+xml
AddOutputFilterByType DEFLATE application/font
AddOutputFilterByType DEFLATE application/font-truetype
AddOutputFilterByType DEFLATE application/font-ttf
AddOutputFilterByType DEFLATE application/font-otf
AddOutputFilterByType DEFLATE application/font-opentype
AddOutputFilterByType DEFLATE application/font-woff
AddOutputFilterByType DEFLATE application/font-woff2
AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
AddOutputFilterByType DEFLATE font/ttf
AddOutputFilterByType DEFLATE font/otf
AddOutputFilterByType DEFLATE font/opentype
AddOutputFilterByType DEFLATE font/woff
AddOutputFilterByType DEFLATE font/woff2
# GZip Compression
BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
</IfModule>
## END JSPEED OPTIMIZATIONS

# php -- BEGIN cPanel-generated handler, do not edit
# Set the “ea-php74” package as the default “PHP” programming language.
<IfModule mime_module>
  AddHandler application/x-httpd-ea-php74 .php .php7 .phtml
</IfModule>
# php -- END cPanel-generated handler, do not edit

Standard in the meaning they would not relate to blocking xml ;)

Oh ok yes, I see now the updated joomla SEF section has changed quite a bit since. I will test and let you know the result, thanks for your support.

Well no, that's not a customisation it's a Joomla Core SEF directive, albeit an old configuration as you pointed out.

Awesome, that's fixed it. Thanks again for your support.