Manticore Search

Manticore Search is a full-text search server; the search plugin talks to it with the manticore driver of _class('wrapper_search'). The admin page Tools / Manticore (?object=manage_manticore) shows how the server is doing and holds the everyday management: the tables and their size, optimize, flush, truncate, drop, creating the tables the site needs, a query console, the threads with kill, the variables with the risky ones marked, and the replication clusters. The engine is _class('admin_manticore') (yf_admin_manticore, plugin search). No client library is used: the page speaks SQL to Manticore over the MySQL protocol (port 9306) with the PHP mysqli extension.

Which server

The page uses the search plugin's server, first match wins:

  1. $PROJECT_CONF['admin_manticore']['HOST'] (for the page only),
  2. $PROJECT_CONF['search_driver_manticore']['HOST'],
  3. the SPHINX_HOST / SPHINX_PORT constants (Manticore is a drop-in for Sphinx),
  4. 127.0.0.1:9306.
$PROJECT_CONF['wrapper_search']['driver'] = 'manticore';
$PROJECT_CONF['search_driver_manticore']['HOST'] = '127.0.0.1:9306';

The menu item shows while the site names a Manticore server (one of the settings above, or the manticore search driver). The page first checks that the port answers, so a stopped server costs one connect timeout (2 s) and no PHP warning. Error messages never show passwords or user:password@ parts.

The page

  • Overview: uptime and version, queries (per second, average time, searches and writes), connections and the refused ones (maxed_out), the thread pool (busy workers, clients, queue, load), the query cache and the tables by type. The ⟳ button refreshes the cards every 5 to 60 seconds; it is off until you choose an interval. Below: the components (SHOW VERSION), the checks, the site's index definitions that have no table, and clusters that are not healthy.
  • Tables: every table (SHOW TABLES) with its type (rt, plain, percolate, distributed), documents, size (RAM and disk), RAM chunk and disk chunks, read from SHOW TABLE ... STATUS; sort by any column, filter by name, 50 a page. A warning sign marks a table with many disk chunks or many deleted documents (optimize it). A table's page shows its status, its settings (SHOW TABLE ... SETTINGS), its columns (DESCRIBE) and SHOW CREATE TABLE.
  • Console: one SQL statement at a time, results in a table with the time it took and SHOW META after a SELECT.
  • Threads: SHOW THREADS (the idle ones too on request) with Kill on each connection.
  • Variables: SHOW VARIABLES and SHOW SETTINGS (the configuration searchd runs with), the risky values highlighted.
  • Cluster (only with replication): each cluster from SHOW STATUS LIKE 'cluster%': status, size, node state, its tables, nodes and queues; a node that is not in the primary component or not synced is reported.

Table actions

ActionStatementTablesConfirm
OptimizeOPTIMIZE TABLE t: merges the disk chunks in the background and frees deleted documentsrta click and a confirm
Flush RAM chunkFLUSH RAMCHUNK t: writes the RAM chunk to a new disk chunkrta click and a confirm
TruncateTRUNCATE TABLE t (cluster:t for a replicated table): deletes every document, keeps the tablert, percolatethe table name typed again
DropDROP TABLE t: off unless ALLOW_DROP; a replicated table must leave its cluster firstallthe table name typed again
CreateCREATE TABLE from the site's definition, through the search driver's create_index()missing onesa click and a confirm
KillKILL id of a connection (the ConnID of SHOW THREADS)-a click and a confirm

Table names must match [A-Za-z_][A-Za-z0-9_]* (at most 64 characters) before they reach a statement, and are sent in backticks. Every change is a POST with the session token; a plain link (GET) only shows a confirm form, a script (ajax) gets 403. The table and the connection id are read from the POST body only, never from the link. Each change writes a row to the admin wall (Logs / Admin / Activity) naming the table and the server, before it runs; a failed one writes a second row with the error.

The site's index definitions

List the indexes the site needs in wrapper_search, with the schema of create_index(). Search does not read it; the Tables tab and the overview offer Create for each one the server lacks, and the doctor reports them.

$PROJECT_CONF['wrapper_search']['INDEXES'] = [
    'products' => ['fields' => ['title' => 'text', 'body' => 'text', 'cat_id' => 'int', 'price' => 'float', 'tag' => 'keyword'], 'suggest' => true],
];

Query console

Reads run at once: SELECT, SHOW, DESCRIBE / DESC, EXPLAIN, CALL KEYWORDS, CALL SUGGEST, CALL QSUGGEST, CALL SNIPPETS and CALL PQ. Anything else is a write. Writes are on (CONSOLE_WRITE, the default) while the page is not read-only: the page shows the statement and runs it only after you confirm, and writes it to the admin wall first. DROP also needs ALLOW_DROP. With CONSOLE_WRITE off the console runs reads only.

  • One statement a run: comments are taken out, one trailing ; is fine, a ; between two statements is refused. Strings are single-quoted; a double quote outside a string is refused (write a phrase inside the string: MATCH('"red boots"')).
  • Server files: a statement that mentions load_files (CALL SNIPPETS with load_files or load_files_scattered reads any file the Manticore server can read) is refused unless ALLOW_LOAD_FILES is on; then it is a write (confirmed and audited).
  • Rows: at most CONSOLE_MAX_ROWS (1000) are shown; a read with a larger LIMIT, a subquery's too, is refused. Without a LIMIT Manticore returns 20 rows.
  • Time: a SELECT without max_query_time gets OPTION max_query_time=10000 (CONSOLE_TIMEOUT_MS), except with FACET; a max_query_time of 0 or above it is lowered to it.
  • Every value is shown as text; long values are cut to 1000 characters.

Checks

CheckWhy it matters
searchd.listen on a public address (warning) or on every interface (notice)Manticore has no passwords: whoever reaches the port can read, change and drop every table. Listen on 127.0.0.1 or a private address, or firewall the ports.
binlog_path emptyWithout the binary log, RT documents written since the last flush are lost on a crash.
auto_optimize=0Disk chunks pile up and searches get slower.
log_level debugA lot of log output.
secondary_indexes=0Attribute filters scan every document.
maxed_out above 0Connections were refused because every worker was busy.
Disk chunks above twice optimize_cutoff, 20 % or more deleted documents (Tables tab)Optimize the table.

The site doctor (group Database) runs these checks too while a Manticore server is configured: the server answers (critical when the site search uses Manticore), its version (Manticore 6 or newer; a Sphinx server is reported), the tables of the index definitions, and the risky settings. The admin home has a Manticore Search card for super-admins.

Access and settings

Super-admins (group 1) only; a block rule does not open the page to other groups. Settings page, tab Search:

$PROJECT_CONF['manage_manticore'] = [
    'READ_ONLY' => false,          // true: no change at all (the read-only console still works)
    'ALLOW_DROP' => false,         // offer DROP TABLE (name typed again) and DROP in the console write mode
    'CONSOLE_WRITE' => true,       // the console may run writes, each confirmed first; false: reads only
    'ALLOW_LOAD_FILES' => false,   // allow load_files (CALL SNIPPETS reading server files) as a console write
    'CONSOLE_MAX_ROWS' => 1000,
    'CONSOLE_TIMEOUT_MS' => 10000, // max_query_time added to a SELECT, and its cap; 0 = none
    'PER_PAGE' => 50,              // tables a page
    'REFRESH_DEFAULT' => 0,        // seconds between card refreshes until an admin chooses (0 = off)
    'MENU_ITEM_NEEDS_CONFIG' => true, // false: the menu item always shows
];
$PROJECT_CONF['admin_manticore'] = [
    'HOST' => '',                  // host:port for this page only; '' = the search plugin's
    'CONNECT_TIMEOUT' => 2,
    'TABLES_STATUS_MAX' => 300,    // tables whose status is read for the list
];

The page works with Sphinx 3 in part (the status, the console, the threads); it says so when the server is not Manticore. The HTTP JSON API (port 9308) is not used.