How to Configure the LiteSpeed Cache Crawler in WordPress
Updated 6 days ago
The LiteSpeed Cache Crawler scans your site's sitemap and regenerates pages whose cache has expired, so visitors receive fewer uncached pages. We enable it at the server level on our LiteSpeed shared, reseller, and cloud hosting plans; in WordPress, you just need to turn it on, provide a sitemap, and run it.
Before you start
You need WordPress administrator access and the LiteSpeed page cache must already be working, as verified by the x-litespeed-cache header showing hit. If you haven't configured it yet, follow How to Configure LiteSpeed Cache in WordPress. You'll also need the URL of your XML sitemap, such as the one generated by your SEO plugin.
Enable the Crawler and add the sitemap
In the WordPress dashboard, go to LiteSpeed Cache → Crawler. In Spanish-language WordPress, the menu appears as Rastreador, with the tabs Summary, Map, Blocklist, and Settings.
On the Settings tab, paste the full URL of your XML sitemap or sitemap index into Custom Sitemap, one per line, and save. Without a configured sitemap, the manual run button is disabled.
On the same tab, enable the Crawler option and save your changes. It is disabled by default.
Adjust the frequency and load
Two settings on that same tab control when crawling runs and how much load it generates. For the first test, you can leave them at their default values.
Crawl Interval. How often the entire sitemap is crawled again. The default is 302400 seconds (three and a half days); LiteSpeed recommends adjusting it based on how long it takes to complete a crawl of your site.
Server Load Limit. The server load limit at which the crawler pauses; the default is 1. The crawler reduces its threads to stay below the limit and stops running if it can't stay within it, even when using just one thread. At 0, the crawler does not run.
The other classic settings can no longer be edited in the interface starting with version 7 of the plugin: the delay between requests (500 microseconds), the interval between runs (600 seconds), the maximum run time (900 seconds), and the page timeout (30 seconds). They can be adjusted using constants such as LITESPEED_CRAWLER_USLEEP or LITESPEED_CRAWLER_RUN_INTERVAL in wp-config.php.
Run the first crawl
The crawler is scheduled by WP-Cron, every 600 seconds by default; you can also start it manually so you don't have to wait.
Go to the Crawler's Summary tab.
Click Run manually (Manually run). Two crawlers can't run at the same time: if a scheduled run and a manual run start together, the first one takes priority.
Follow the run details. A URL that's already cached counts as a
hitand the crawler skips it; a URL without a cache counts as amissand the crawler caches it.
To keep runs from depending on site visits, you can move WP-Cron to a system cron job; in cPanel, LiteSpeed recommends using Cron Jobs with a frequency of five minutes.
If the Crawler doesn't start
If the Run manually button is disabled, first check that a URL is saved in Custom Sitemap. If the message "The crawler feature is not enabled on the LiteSpeed server. Please consult your server admin or hosting provider." also appears in Summary, the crawler isn't enabled on the server. Contact us through the support desk at mi.duplika.com, and we'll review your plan's configuration.