Sitemap
The site's /sitemap.xml lists the pages search engines should know about. yf builds it from the sitemap hooks of the
user modules: a module with a public _hook_sitemap($sitemap) (or _hook_site_map(), _site_map_items())
adds its URLs, and the generator (the user module site_map) writes them to files under uploads/site_map/, one set
per language, and serves them for SITEMAP_LIVE_TIME (12 hours) before the next request writes them again. The framework's
news, static_pages, faq and shop have hooks. Generation is off until
site_map.SITE_MAP_ENABLED is on (Settings, tab SEO); the web server routes /sitemap.xml to the
sitemapxml module, which answers an empty sitemap on every host that is not in main()->PRODUCTION_DOMAIN.
The admin page Tools / Sitemap (?object=manage_sitemap) shows what the site has: the modules with a hook and how many
URLs each added, the files per language with their size, age and cache state, the settings, the real files and the site's own answer, and
the problems search engines would find. The admins the block rules let open it see everything (yf_sitemap_manager::$VIEW_GROUPS
narrows it); regenerating, clearing, fetching and IndexNow are for super-admins (group 1), each a POST with the session token, written to the
admin wall.
Adding URLs from a module
public function _hook_sitemap($sitemap)
{
foreach ((array) db()->from('articles')->where('active', '1')->get_all() as $a) {
$sitemap->_add([
'url' => '/articles/show/' . (int) $a['id'], // a path goes through url(); an absolute URL is kept
'last_update' => (int) $a['edit_date'], // lastmod: a timestamp or a date string
]);
}
return true;
}
A string item is just the URL. priority and changefreq may be given too (search engines mostly ignore them;
DEFAULT_PRIORITY and DEFAULT_CHANGEFREQ fill them in). With site_map.EXTENSIONS on, an item may also carry
images (URLs), videos (thumbnail_loc, title, description, content_loc or
player_loc), news (name, language, publication_date, title) and
alternates (['en' => url, 'de' => url, 'x-default' => url], the hreflang versions of the page). Leave out
pages behind a login, pages with noindex and URLs whose canonical points elsewhere. A hook that returns an array of items works
too. Which modules run is site_map.MODULES_TO_INCLUDE (empty: every active module with a hook) minus
site_map.MODULES_EXCLUDE (names or globs).
Overview
Every module with a hook (read from the files, as on the User modules page) and its state: included, inactive (the user module is off),
excluded, not listed (a module list is set without it). While the admin module manage_sitemap is active (or
$PROJECT_CONF['site_map']['REPORT_DIR'] names a folder; '' turns it off), each generation writes a report to
STORAGE_PATH/sitemap/: the URLs,
skipped items and milliseconds per module, the files and the time it took. A generation that dies in a hook leaves its report "running" with
the module's name, which the page and the doctor show. The files table says per language how many files and URLs there are, when they were
written and whether the cache is fresh or will be rebuilt by the next request.
Regenerate deletes the cached files and requests the site's own sitemap URL, so the files are built where they always are: in a
request of the public site (only there do url() and the modules' hooks make public links). On production that is
/sitemap.xml; elsewhere ?object=site_map of the site URL. The site URL is the trusted base URL of the SEO page and
the doctor ($PROJECT_CONF['seo']['BASE_URL'], or WEB_PATH when its host is in DOMAINS); without one,
Regenerate only clears the cache and the next visitor's request builds it. Clear cache only deletes the files. Both are refused
while a generation holds the generator's lock (USE_LOCKING, a lock file younger than LOCK_TIMEOUT). Fetch as a search
engine requests the sitemap (gzip accepted, no cookies) and keeps the status, headers, size and the start of the answer for the Preview
tab.
Preview and validation
Preview shows a generated file as it is served (escaped, with line numbers, page by page) or as a table of its URLs. Validation reads the
files of a language without network access and without entities (a DOCTYPE is refused): well-formed XML, the urlset /
sitemapindex root and namespace, at most 50000 URLs and 50 MB per file, several files without an index, index entries without
a file, duplicates, URLs that are relative, longer than 2048 characters, on another host or scheme, or not canonical in shape (a fragment,
session or tracking parameters such as utm_* or PHPSESSID, index.php, //, an upper-case host),
login and action pages (yf_seo::$PATH_DENY), lastmod format, lastmod in the future, one lastmod for every URL, no lastmod at all,
changefreq and priority values. When the SEO page has a scan, its results add the URLs that answered something else than 200, redirected,
were noindex, named another canonical URL or are blocked by robots.txt; nothing is requested for that.
Settings and switches
All on the Settings page, tab SEO (the Sitemap page shows them with their source). Besides the module lists, the defaults and the limits
(MAX_ENTRIES URLs and MAX_SIZE bytes per file, SITEMAP_LIVE_TIME, USE_LOCKING), five
switches change the output, each off so that a site keeps its sitemap byte for byte until it turns one on:
site_map.FILES_FIX: the URLs are XML-escaped (a URL with&made the old file invalid XML), relative./?URLs made absolute, several files joined by a sitemap index (before, only the first file was served), files written under a temporary name and renamed when complete, only the language being generated replaced, and a request that finds another generation running gets the previous file.site_map.HTTP_HEADERS:application/xml; charset=UTF-8,Last-Modifiedwith a304answer toIf-Modified-Since,Cache-Controlfor the rest of the cache lifetime andX-Robots-Tag: noindex(the XML file itself stays out of search results; search engines still read it).site_map.GZIP: gzip to clients that accept it (a.gzcopy next to the file); not needed when the web server compresses XML.site_map.LASTMOD_FROM_DATA: the framework's hooks give the real date of their content as lastmod (news: its date).site_map.EXTENSIONS: image, video, news and hreflang entries of hook items, with their namespaces.
robots.txt names the sitemap with robotstxt.SITEMAP_LINE (see SEO). NOTIFY_GOOGLE pings an endpoint Google
retired in 2023; the doctor asks to turn it off.
IndexNow
IndexNow tells Bing, Yandex, Seznam, Naver and Yep (not Google) which URLs changed. It is off (sitemap_manager.INDEXNOW_ALLOW,
Settings tab SEO, super-admin; turning it on there needs a typed confirmation, and each submission is a button the browser asks to confirm). The site proves it owns its host with a key file in the web root, named and holding the
key (8 to 128 characters of letters, digits and -): the IndexNow tab creates a random key and writes the file (never over a link
or a file with other content), or the site config sets the key. Remove the key file deletes it again, only while it holds the
current key and is not a link (also with IndexNow off):
$PROJECT_CONF['sitemap_manager']['INDEXNOW_KEY'] = 'a1b2c3d4e5f6a7b8'; // and the file a1b2c3d4e5f6a7b8.txt in the web root
Only a super-admin's button (Submit, or Regenerate when IndexNow is ready) sends a request, only on a production site, only URLs of the
site's own host, only the ones not sent before or whose lastmod changed, at most INDEXNOW_BATCH (10000) per request, to
INDEXNOW_ENDPOINT (https://api.indexnow.org/indexnow, which shares them with every participating engine; the
endpoint must be https on one of INDEXNOW_HOSTS). The request goes over the doctor's socket client (TLS verified, deadline, byte
cap, no redirect followed).
Index status
Whether a page is indexed only the search engines know. Google Search Console (URL Inspection and the Sitemaps report) and Bing Webmaster Tools show it; their APIs need a verified site and a service account or an API key, and are not built in yet. The SEO page shows whether each URL can be indexed (answers 200, no noindex, canonical to itself, not blocked by robots.txt).
Elsewhere
The site doctor's SEO group checks that the last generation finished, the files are valid (the file fix as a quick fix when several files have no index or the XML is invalid), the retired Google ping is off and IndexNow is ready when it is on. The admin home has a Sitemap card. The SEO page links here and checks the sitemap the way search engines fetch it.