Sitemap

The site's /sitemap.xml lists the pages search engines should know about. yf builds it from the sitemap hooks of the user modules: a module with a public _hook_sitemap($sitemap) (or _hook_site_map(), _site_map_items()) adds its URLs, and the generator (the user module site_map) writes them to files under uploads/site_map/, one set per language, and serves them for SITEMAP_LIVE_TIME (12 hours) before the next request writes them again. The framework's news, static_pages, faq and shop have hooks. Generation is off until site_map.SITE_MAP_ENABLED is on (Settings, tab SEO); the web server routes /sitemap.xml to the sitemapxml module, which answers an empty sitemap on every host that is not in main()->PRODUCTION_DOMAIN.

The admin page Tools / Sitemap (?object=manage_sitemap) shows what the site has: the modules with a hook and how many URLs each added, the files per language with their size, age and cache state, the settings, the real files and the site's own answer, and the problems search engines would find. The admins the block rules let open it see everything (yf_sitemap_manager::$VIEW_GROUPS narrows it); regenerating, clearing, fetching and IndexNow are for super-admins (group 1), each a POST with the session token, written to the admin wall.

Adding URLs from a module

public function _hook_sitemap($sitemap)
{
    foreach ((array) db()->from('articles')->where('active', '1')->get_all() as $a) {
        $sitemap->_add([
            'url' => '/articles/show/' . (int) $a['id'], // a path goes through url(); an absolute URL is kept
            'last_update' => (int) $a['edit_date'],      // lastmod: a timestamp or a date string
        ]);
    }
    return true;
}

A string item is just the URL. priority and changefreq may be given too (search engines mostly ignore them; DEFAULT_PRIORITY and DEFAULT_CHANGEFREQ fill them in). With site_map.EXTENSIONS on, an item may also carry images (URLs), videos (thumbnail_loc, title, description, content_loc or player_loc), news (name, language, publication_date, title) and alternates (['en' => url, 'de' => url, 'x-default' => url], the hreflang versions of the page). Leave out pages behind a login, pages with noindex and URLs whose canonical points elsewhere. A hook that returns an array of items works too. Which modules run is site_map.MODULES_TO_INCLUDE (empty: every active module with a hook) minus site_map.MODULES_EXCLUDE (names or globs).

Overview

Every module with a hook (read from the files, as on the User modules page) and its state: included, inactive (the user module is off), excluded, not listed (a module list is set without it). While the admin module manage_sitemap is active (or $PROJECT_CONF['site_map']['REPORT_DIR'] names a folder; '' turns it off), each generation writes a report to STORAGE_PATH/sitemap/: the URLs, skipped items and milliseconds per module, the files and the time it took. A generation that dies in a hook leaves its report "running" with the module's name, which the page and the doctor show. The files table says per language how many files and URLs there are, when they were written and whether the cache is fresh or will be rebuilt by the next request.

Regenerate deletes the cached files and requests the site's own sitemap URL, so the files are built where they always are: in a request of the public site (only there do url() and the modules' hooks make public links). On production that is /sitemap.xml; elsewhere ?object=site_map of the site URL. The site URL is the trusted base URL of the SEO page and the doctor ($PROJECT_CONF['seo']['BASE_URL'], or WEB_PATH when its host is in DOMAINS); without one, Regenerate only clears the cache and the next visitor's request builds it. Clear cache only deletes the files. Both are refused while a generation holds the generator's lock (USE_LOCKING, a lock file younger than LOCK_TIMEOUT). Fetch as a search engine requests the sitemap (gzip accepted, no cookies) and keeps the status, headers, size and the start of the answer for the Preview tab.

Preview and validation

Preview shows a generated file as it is served (escaped, with line numbers, page by page) or as a table of its URLs. Validation reads the files of a language without network access and without entities (a DOCTYPE is refused): well-formed XML, the urlset / sitemapindex root and namespace, at most 50000 URLs and 50 MB per file, several files without an index, index entries without a file, duplicates, URLs that are relative, longer than 2048 characters, on another host or scheme, or not canonical in shape (a fragment, session or tracking parameters such as utm_* or PHPSESSID, index.php, //, an upper-case host), login and action pages (yf_seo::$PATH_DENY), lastmod format, lastmod in the future, one lastmod for every URL, no lastmod at all, changefreq and priority values. When the SEO page has a scan, its results add the URLs that answered something else than 200, redirected, were noindex, named another canonical URL or are blocked by robots.txt; nothing is requested for that.

Settings and switches

All on the Settings page, tab SEO (the Sitemap page shows them with their source). Besides the module lists, the defaults and the limits (MAX_ENTRIES URLs and MAX_SIZE bytes per file, SITEMAP_LIVE_TIME, USE_LOCKING), five switches change the output, each off so that a site keeps its sitemap byte for byte until it turns one on:

  • site_map.FILES_FIX: the URLs are XML-escaped (a URL with & made the old file invalid XML), relative ./? URLs made absolute, several files joined by a sitemap index (before, only the first file was served), files written under a temporary name and renamed when complete, only the language being generated replaced, and a request that finds another generation running gets the previous file.
  • site_map.HTTP_HEADERS: application/xml; charset=UTF-8, Last-Modified with a 304 answer to If-Modified-Since, Cache-Control for the rest of the cache lifetime and X-Robots-Tag: noindex (the XML file itself stays out of search results; search engines still read it).
  • site_map.GZIP: gzip to clients that accept it (a .gz copy next to the file); not needed when the web server compresses XML.
  • site_map.LASTMOD_FROM_DATA: the framework's hooks give the real date of their content as lastmod (news: its date).
  • site_map.EXTENSIONS: image, video, news and hreflang entries of hook items, with their namespaces.

robots.txt names the sitemap with robotstxt.SITEMAP_LINE (see SEO). NOTIFY_GOOGLE pings an endpoint Google retired in 2023; the doctor asks to turn it off.

IndexNow

IndexNow tells Bing, Yandex, Seznam, Naver and Yep (not Google) which URLs changed. It is off (sitemap_manager.INDEXNOW_ALLOW, Settings tab SEO, super-admin; turning it on there needs a typed confirmation, and each submission is a button the browser asks to confirm). The site proves it owns its host with a key file in the web root, named and holding the key (8 to 128 characters of letters, digits and -): the IndexNow tab creates a random key and writes the file (never over a link or a file with other content), or the site config sets the key. Remove the key file deletes it again, only while it holds the current key and is not a link (also with IndexNow off):

$PROJECT_CONF['sitemap_manager']['INDEXNOW_KEY'] = 'a1b2c3d4e5f6a7b8'; // and the file a1b2c3d4e5f6a7b8.txt in the web root

Only a super-admin's button (Submit, or Regenerate when IndexNow is ready) sends a request, only on a production site, only URLs of the site's own host, only the ones not sent before or whose lastmod changed, at most INDEXNOW_BATCH (10000) per request, to INDEXNOW_ENDPOINT (https://api.indexnow.org/indexnow, which shares them with every participating engine; the endpoint must be https on one of INDEXNOW_HOSTS). The request goes over the doctor's socket client (TLS verified, deadline, byte cap, no redirect followed).

Index status

Whether a page is indexed only the search engines know. Google Search Console (URL Inspection and the Sitemaps report) and Bing Webmaster Tools show it; their APIs need a verified site and a service account or an API key, and are not built in yet. The SEO page shows whether each URL can be indexed (answers 200, no noindex, canonical to itself, not blocked by robots.txt).

Elsewhere

The site doctor's SEO group checks that the last generation finished, the files are valid (the file fix as a quick fix when several files have no index or the XML is invalid), the retired Google ping is off and IndexNow is ready when it is on. The admin home has a Sitemap card. The SEO page links here and checks the sitemap the way search engines fetch it.