SEO

The admin page Tools / SEO (?object=manage_seo) looks at the site the way a search engine does: it requests the site's own pages over HTTP, reads what they say to search engines and lists what to improve, each finding with a severity (critical, warning, notice), the pages it concerns and how to fix it. Safe fixes can be applied from the page, one tick each, with a ledger and an undo. Nothing changes on a site until a super-admin acts: the page, the doctor checks and the admin home card only read the result of the last scan.

Who. Every admin the block rules let open manage_seo sees the last scan, its pages, the performance results, the fixes on offer and the fix ledger (yf_seo::$VIEW_GROUPS narrows it to some admin groups). Scans, fixes, undo, Lighthouse and PageSpeed Insights are for super-admins (group 1) only.

Scanning

  • What is requested. The home page, the URLs of /sitemap.xml, then the links found, breadth first, up to the chosen number of pages (10 to 500, default yf_seo::$MAX_PAGES 50) and within TIME_BUDGET_MS (20 s); a scan stopped by a limit says so. A gzip sitemap may unpack to 20 MB at most (GUNZIP_MAX); a sitemap with a NUL byte, a UTF-16 or UTF-32 byte order mark (a UTF-8 one is fine) or a DOCTYPE is refused. robots.txt, the sitemap, an unknown URL (soft 404), http:// of an https site, the www / non-www twin and one page with and without its trailing slash are asked too.
  • Only this site. The site URL is $PROJECT_CONF['seo']['BASE_URL'], else the trusted base URL of the doctor's network checks (their BASE_URL, or WEB_PATH when the site sets it or its host is in DOMAINS; never the Host header of a request alone). Links to other hosts are counted, never requested; more host names of the site go in yf_seo::$HOSTS.
  • Like a visitor. No cookies, no session, only GET, the User-Agent yf-seo. Links whose path or query match yf_seo::$PATH_DENY (logout, delete, vote, cart, ...), the admin path, files (images, PDF, archives) and rel="nofollow" links are not followed; neither is a redirect to a denied path, a sitemap URL on a denied path or an asset on a denied path (a pattern that fails counts as denied).
  • The one exception to "this host only" is the www / non-www twin of the base host, asked once without following, and only when the host name is public (not an IP address, not localhost, .local, .test, .internal ...) and the twin resolves to public addresses only (no private, loopback or link-local address); the request goes to the address that was checked.
  • Bounded. Every request has a hard time limit (NET_TIMEOUT_MS) and a byte cap (MAX_BYTES); redirects are followed to this site only (MAX_REDIRECTS); one scan runs at a time.
  • Cached. The result (findings and a compact record of each page, no page bodies) is kept in STORAGE_PATH/seo/ with the scan before it, so the page shows what is new and what is gone. The folder gets a deny-all .htaccess, which only Apache reads: on nginx (or any other server) STORAGE_PATH must lie outside the web root or be denied in the server config, as for the doctor's reports.

Indexing and robots

A production page with noindex (robots meta tag or X-Robots-Tag) is critical; a development or staging copy that search engines may index is a warning (production is the doctor's environment setting). robots.txt: missing, answered with an HTML page, Disallow: / on production, no Sitemap: line, sitemap URLs that robots.txt blocks. The framework's robots.txt is the robotstxt module: it allows indexing only on the hosts of main()->PRODUCTION_DOMAIN; the setting robotstxt.SITEMAP_LINE (Settings, tab SEO, off by default) adds the Sitemap: line there.

Sitemap

Missing, not XML, empty, URLs of another host, URLs that do not answer 200. The framework's sitemap is the site_map module (site_map.SITE_MAP_ENABLED, Settings tab SEO, off by default): modules add their pages with _hook_sitemap().

Titles, descriptions and headings

Missing, several, too short or too long, the same on several pages (pages whose canonical names another URL are left out). A module sets the title with _hook_title() and the meta tags with _hook_meta() (keys description, canonical, og_title ...); static pages and news have their own fields. One <h1> per page.

Canonical URLs and languages

Canonical: missing (notice), several, to another host, to a page that does not answer 200. A multi-language site (pages in several languages, ?lang= links or any hreflang) is checked for hreflang links: missing, invalid codes, no x-default, no link to itself, a target that does not link back.

Internal links to pages that answer an error, links to redirects, redirect chains and loops, pages that answer 5xx, and the soft 404: an unknown URL that answers 200 (turn on core_errors.ERROR_PAGES or let the site answer 404).

A public site gets requests from scanners for paths it never had (/wp-admin/..., /.env). Each one answers 404 and, by default, also writes the warning yf_core_blocks: Task not found to the error log, where it hides real errors. Set $PROJECT_CONF['graphics']['NOT_FOUND_RAISE_WARNING'] = false; in the site config to answer the 404 without the log row.

Images, social cards and structured data

Images without an alt attribute (alt="" is right for a decorative image; the page view suggests text from the file names), without width and height, all loaded at once. OpenGraph (og:title, og:description, og:image) and twitter:card. JSON-LD blocks must be JSON with @context and @type; the home page should have one (Organization or WebSite).

HTTPS and hosts

The site URL on http, http not redirecting to https, mixed content (http scripts and styles on an https page are critical), both www and non-www answering, a page answering with and without the trailing slash.

Language and content

No <html lang>, no viewport meta tag, pages with the same text under several URLs, pages with fewer than 50 words.

Performance

Lite audit (every scan, no browser): the home page's HTML size and compression, its requests (assets of other hosts are counted, not fetched), the weight with the assets of this site, render-blocking scripts and stylesheets in <head>, images over 200 KB, text answers without gzip / brotli, static assets without cache headers, slow answers, inline code; per page the HTML size and the blocking assets.

Lighthouse (Apache-2.0, Google): the paint metrics and scores of a real browser. It needs Node, Chrome or Chromium and the lighthouse program on the web server (npm install -g lighthouse); the framework never installs it. Chrome loads the page like a visitor's browser: it runs the page's JavaScript, third-party scripts included, with network access from the web server. Turn it on only for a site whose pages you trust. The Performance tab says what it finds (a file lookup; nothing is run for that). A super-admin may run it on the site URL or a scanned page when seo.LIGHTHOUSE_ALLOW is on (Settings, tab SEO): it runs as an argument list without a shell, with a timeout (LIGHTHOUSE_TIMEOUT) after which the whole process group (Chrome too) is killed (with setsid and the posix extension; without them GNU timeout signals the group, and without either only the program itself is stopped), one run at a time, and keeps the scores and main metrics of the last runs. LIGHTHOUSE_BIN, CHROME_PATH and CHROME_FLAGS set the programs and the browser flags (no --no-sandbox unless the server needs it).

PageSpeed Insights (Google API, opt-in): Google runs Lighthouse on its own servers and adds the speed real visitors had (Chrome UX Report). The URL of the page is sent to Google. It is off by default (seo.PAGESPEED_ALLOW, Settings tab SEO, super-admin, typed confirmation) and needs an API key in the site config, never saved by a page and shown masked:

$PROJECT_CONF['seo']['PAGESPEED_KEY'] = 'AIza...'; // Google Cloud console: PageSpeed Insights API

Only a super-admin's button sends a request, only for a production site, and only a URL of the site's own public host name (the base URL or a scanned page); the key never goes into the results, messages or the audit. PAGESPEED_STRATEGY picks mobile or desktop.

Fixes

The Fixes tab lists what can be fixed now, each with its own tick; nothing is applied without it.

  • Settings the scan says are missing on a production site (the robots.txt Sitemap line, sitemap generation): saved through the Settings page's save (its checks and backup), as the doctor's quick fixes are; settings for super-admins only or with a typed confirmation stay on their own pages.
  • robots.txt file: when robots.txt is a static file in the web root, the Sitemap: line is appended to it (not a symbolic link, at most 64 KB) under a lock, so two requests never mix; the previous text is kept for the undo. The file is written as a new file and renamed over the old one: its mode stays, but it then belongs to the PHP user, and ACLs or extended attributes of the old file are not kept.
  • Empty meta fields of static pages (meta description, page title) and news (meta description): a suggestion from the row's own text (tags and braces taken out, cut at a word near 155 characters), which can be edited before applying. A field is written only while it is still empty (the condition is part of the UPDATE): a value never gets overwritten. Fields with nothing to suggest from take typed text.

Every fix goes to the fix ledger (who, when, what before, what after) and the admin wall. Undo puts the value back only while it is still what the fix wrote: a field or file edited since is kept.

Safety

  • Viewing for the admins the block rules let in; every scan, fix, undo, Lighthouse and PageSpeed Insights run is for super-admins (group 1), a POST with the session token.
  • Crawled text (titles, descriptions, URLs) is shown escaped, template braces included.
  • Content writes use whitelisted tables and columns (yf_seo::$META_TARGETS), integer ids and escaped values.
  • XML is parsed without network access and refused with a DOCTYPE; HTML is parsed with DOMDocument, which loads nothing.

Elsewhere

The site doctor has an SEO group: a production site with ROBOTS_NO_INDEX on (critical), the problems of the last scan, and the two settings above as quick fixes. The admin home has an SEO card with the score of the last scan.