diff --git a/AGENTS.md b/AGENTS.md index 3ba9b23..7319f00 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ - **trellis/** - Trellis-specific: `backup/`, `monitoring/`, `updater/` - **wp-cli/** - WordPress CLI tools: `content-creation/`, `diagnostics/`, `migration/` - **scripts/** - Utilities: `backup/`, `monitoring/`, `git/create-pr.sh`, `sync/rsync-theme.sh` -- **docs/** - Design docs at the top level; guides in subdirectories: `nginx/` (server configs), `troubleshooting/` (server/WP), `bedrock/` (Composer workflows), `wordpress-utilities/` (copy-paste components) +- **docs/** - Design docs at the top level; guides in subdirectories: `nginx/` (server configs), `bedrock/` (Composer workflows), `wordpress-utilities/` (copy-paste components) - **Root docs**: `README.md`, `CLAUDE.md`, `CREATE-PR.md`, `AGENTS.md`, `CHANGELOG.md`, `LICENSE.md` - Keep new tools self-contained: add to appropriate category with a concise `README.md` and example configs. diff --git a/CHANGELOG.md b/CHANGELOG.md index 51aff89..dc07931 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Removed +- **The troubleshooting guides**, now covered on wpvilla.in: [PHP-FPM memory exhaustion](https://wpvilla.in/debugging-php-fpm-memory-exhaustion-wordpress-woocommerce-trellis/), [mail that never arrives or bounces](https://wpvilla.in/trellis-mail-not-sending-smtp-config-reset-and-bounces/), [the MariaDB failure at "Set root user password"](https://wpvilla.in/trellis-provision-set-root-user-password-fails-mariadb-provider-plugins/) and a new post on [the OOM killer, cron and the unlimited PHP CLI memory limit](https://wpvilla.in/linux-oom-killer-wordpress-cron-php-cli-memory-limit-trellis/). `docs/troubleshooting/` is gone, and the README, `AGENTS.md` and `CLAUDE.md` no longer point at it. The generic MariaDB diagnostics and the PHP-FPM sizing tables are not carried over; they stay in git history. - **The security guides**, now four posts on wpvilla.in: [the two scanners and checksum verification](https://wpvilla.in/wordpress-malware-scanners-targeted-general-checksum-verification/), [reading scan results](https://wpvilla.in/reading-malware-scan-results-false-positives-wordpress-hardening/), [a permanent Nginx deny list](https://wpvilla.in/nginx-deny-list-trellis-cidr-whitelist-self-lockout/) and [a Trellis security routine](https://wpvilla.in/trellis-wordpress-security-routine-abuseipdb-fail2ban/), plus the existing [fail2ban post](https://wpvilla.in/fail2ban-wordpress-wp-login-trellis-nginx-deny-list/). The `README`, `SECURITY-GUIDE` and `SCANNER-SUMMARY` guides in `wp-cli/security/` and the `README`, `FAIL2BAN` and `MANUAL-IP-BLOCKING` guides in `trellis/security/` are gone; the scanners and the AbuseIPDB scripts stay. The scanner wrapper's closing hint now names the post. - **The provisioning and updater guides**, now four posts on wpvilla.in: [a new Mac for Trellis](https://wpvilla.in/new-mac-trellis-development-setup-lima-composer-node/), [cloning an existing project](https://wpvilla.in/clone-trellis-bedrock-project-lima-vm-first-provision/), [updating Trellis](https://wpvilla.in/update-trellis-without-losing-customisations-rsync-excludes/) and [provision tags and the PHP upgrade](https://wpvilla.in/trellis-provision-tags-php-version-upgrade-checklist/). The cron guide is covered by the existing [WP-Cron interval post](https://wpvilla.in/trellis-wp-cron-interval-scheduled-posts-late-cron-interval/). `trellis/provision/` is gone, as are the updater README and `manual-update.md`; the `trellis-updater` script stays. - **The content-creation guides**, now three posts on wpvilla.in: [creating pages from block markup](https://wpvilla.in/create-wordpress-pages-block-markup-wp-cli-trellis-vm-production/), [block patterns as PHP files](https://wpvilla.in/wordpress-block-pattern-php-files-headers-escaping-images/) and [pattern layout rules](https://wpvilla.in/wordpress-block-pattern-layout-alignfull-grid-spacing-rules/). `PAGE-CREATION.md`, `PATTERN-REQUIREMENTS.md` and the folder README are gone; the scripts and the example stay. diff --git a/CLAUDE.md b/CLAUDE.md index e68ac57..e47bc14 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,6 @@ This is a collection of tools, scripts, and documentation for WordPress operatio - `docs/dev/` - Design and planning docs for wp-ops itself. New ones go here, not at the `docs/` top level - `docs/cli/` - How the `wp-ops` binary works: usage reference (categories, platforms, completion, guide search) and the trellis-cli plugin - `docs/nginx/` - Web server configs: `browser-caching/`, `image-optimization/`, `redirects/` (guides plus the `.conf.j2` templates they describe) - - `docs/troubleshooting/` - Server and WordPress troubleshooting guides (OOM, PHP-FPM, MariaDB, mail) - `docs/bedrock/` - Bedrock/Composer workflows: local package development, and the standard `wp-cli.yml` for Bedrock path setup - `docs/wordpress-utilities/` - Copy-paste-into-a-project components and reference material: `snippets/`, `speed-optimization/` diff --git a/README.md b/README.md index 555e778..fb29dc8 100644 --- a/README.md +++ b/README.md @@ -136,7 +136,6 @@ sites are run with: | **Content & SEO** | Page creation, block pattern validation, redirect/schema/orphan-page audits | [SEO](wp-cli/seo/README.md) | | **Nginx** | WebP/AVIF serving, browser caching and redirect templates for Trellis `nginx-includes/` | [images](docs/nginx/image-optimization/README.md) · [caching](docs/nginx/browser-caching/README.md) · [redirects](docs/nginx/redirects/README.md) | | **Bedrock** | Testing unreleased packages via Composer `path` repos, standard `wp-cli.yml` | [packages](docs/bedrock/local-package-development/README.md) · [wp-cli.yml](docs/bedrock/wp-cli-config/README.md) | -| **Troubleshooting** | PHP-FPM, MariaDB, OOM and mail diagnostics | [→](docs/troubleshooting/README.md) | | **Scripts & guides** | Theme sync, PR helpers, WP-CLI snippets for Bedrock | [scripts](scripts/README.md) · [snippets](docs/wordpress-utilities/snippets/README.md) | ## Requirements diff --git a/docs/troubleshooting/MAIL.md b/docs/troubleshooting/MAIL.md deleted file mode 100644 index 008d32b..0000000 --- a/docs/troubleshooting/MAIL.md +++ /dev/null @@ -1,366 +0,0 @@ -# Mail Configuration Issues - -The story of both failures and how to tell them apart: [WordPress Mail Not Arriving on Trellis: Two Different Failures That Look the Same](https://wpvilla.in/trellis-mail-not-sending-smtp-config-reset-and-bounces/). - -## SSH User Access - -**Important**: When troubleshooting mail issues, use the appropriate SSH user: - -- **`web` user** (Recommended for most operations) - - Read-only access to logs and files - - Can run WP-CLI commands - - No sudo access (passwordless) - - Example: `ssh web@example.com` - -- **`admin_user`** (For administrative tasks) - - Full sudo access with password - - Required for: viewing system logs, checking protected config files - - Use when commands require `sudo` - - Example: `ssh admin_user@example.com "sudo journalctl"` - - **Replace `admin_user` with your configured username** (e.g., `admin`, `deploy`, `warden`, your name, etc.) - -- **`root` user** (Avoid - limited availability) - - Only available on machines that provisioned the server - - Not accessible from most machines - - Trellis best practice: Don't use root for routine operations - -**Throughout this guide**: Commands use `web@example.com` or `admin_user@example.com` as placeholders. Replace `admin_user` with your server's configured admin username. - ---- - -## Table of Contents - -- [Issue 1: sSMTP Configuration After Trellis Upgrade](#issue-1-ssmtp-configuration-after-trellis-upgrade) - - [Problem](#problem) - - [Symptoms](#symptoms) - - [Diagnosis](#diagnosis) - - [Solution](#solution) - - [Prevention](#prevention) -- [Issue 2: WordPress Email Bounces - Non-Existent Admin Email](#issue-2-wordpress-email-bounces---non-existent-admin-email) - - [Problem](#problem-1) - - [Symptoms](#symptoms-1) - - [Diagnosis](#diagnosis-1) - - [Solution](#solution-1) - - [Testing](#testing) - - [Prevention](#prevention-1) - ---- - -## Issue 1: sSMTP Configuration After Trellis Upgrade - -### Problem - -After a Trellis upgrade, mail functionality stopped working because the main configuration file `trellis/group_vars/all/mail.yml` was overwritten with default settings. - -### Symptoms - -System logs show repeated SMTP connection failures with error messages like: - -- `Unable to locate smtp.example.com` -- `Cannot open smtp.example.com:587` -- `MAIL (mailed X bytes of output but got status 0x0001 from MTA)` - -### Diagnosis - -#### Check System Logs - -SSH into the server and examine recent logs for mail-related errors: - -```bash -ssh admin_user@domain.com "sudo journalctl -b | tail -50" -``` - -Note: Use your admin user (with sudo access) for administrative tasks. The `web` user does not have sudo access. - -#### Key Error Messages to Look For - -``` -sSMTP[12502]: Unable to locate smtp.example.com -cron[12502]: sendmail: Cannot open smtp.example.com:587 -sSMTP[12502]: Cannot open smtp.example.com:587 -CRON[12499]: (root) MAIL (mailed 65 bytes of output but got status 0x0001 from MTA -``` - -#### Verify Mail Configuration - -Check if `trellis/group_vars/all/mail.yml` contains default placeholder values instead of actual SMTP credentials. - -### Solution - -#### Restore Proper Mail Settings - -Edit `trellis/group_vars/all/mail.yml` with your actual SMTP configuration: - -```yml -# Documentation: https://roots.io/trellis/docs/mail/ -mail_smtp_server: smtp.yourdomain.com:587 -mail_admin: admin@yourdomain.com -mail_hostname: yourdomain.com -mail_user: your_smtp_username -mail_password: "{{ vault_mail_password }}" # Define this variable in group_vars/all/vault.yml -``` - -#### Set Vault Password - -Ensure the SMTP password is securely stored in `trellis/group_vars/all/vault.yml`: - -```yml -vault_mail_password: your_actual_smtp_password -``` - -#### Re-provision Server - -Apply the updated mail configuration: - -```bash -trellis provision production -``` - -Or provision only the sSMTP configuration: - -```bash -trellis provision --tags ssmtp production -``` - -### Prevention - -#### Before Upgrading Trellis - -1. **Backup configuration files** - Especially `group_vars/all/mail.yml` -2. **Review upgrade notes** - Check if configuration file formats have changed -3. **Use version control** - Commit current configuration before upgrading -4. **Document custom settings** - Keep a record of non-default values - -#### After Upgrading Trellis - -1. **Review configuration files** - Check for overwritten or reset values -2. **Compare with backup** - Verify custom settings were preserved -3. **Test mail functionality** - Send test emails after provisioning -4. **Monitor logs** - Watch for SMTP errors in the first 24 hours - -#### Recommended Tools - -Check mail configuration on remote server: - -```bash -# Using admin user (requires sudo for protected files) -ssh admin_user@domain.com "sudo cat /etc/ssmtp/ssmtp.conf" -``` - -Test mail sending: - -```bash -echo "Test message" | mail -s "Test Subject" your-email@example.com -``` - ---- - -## Issue 2: WordPress Email Bounces - Non-Existent Admin Email - -### Problem - -WordPress sends emails to the configured admin email address (e.g., site registration notifications, password resets, user notifications). If this email address doesn't exist, emails will bounce with "soft bounce" or "connection closed by recipient's server" errors. - -This is especially common with **WordPress Multisite** where: -- Network admin emails are set to non-existent addresses (e.g., `admin@demo.example.com`) -- New site registrations trigger emails to the network admin -- Subsite admin emails may also be invalid - -**Important**: This issue occurs even when sSMTP is properly configured. The server successfully sends the email through SMTP relay (e.g., Brevo), but the recipient's mail server rejects it because the mailbox doesn't exist. - -### Symptoms - -Email delivery logs or monitoring services (e.g., Brevo dashboard) show: - -- **Soft bounce**: "Connection closed by recipient's server" -- **Hard bounce**: "Mailbox does not exist" or "User unknown" -- Email status: "Deferred" or "Failed" -- Emails successfully leave your server but never arrive - -Example from Brevo logs: -``` -From: wordpress@demo.example.com -To: admin@demo.example.com -Status: Soft bounce -Error: connection closed by recipient's server -``` - -### Diagnosis - -#### Check WordPress Admin Email - -**For single sites:** -```bash -ssh web@example.com "cd /srv/www/example.com/current && \ - wp option get admin_email --path=web/wp" -``` - -**For multisite networks:** -```bash -# Check network-wide admin email -ssh web@example.com "cd /srv/www/example.com/current && \ - wp site option get admin_email --path=web/wp --url=https://example.com" - -# Check all subsites -ssh web@example.com "cd /srv/www/example.com/current && \ - for site in \$(wp site list --field=url --path=web/wp --url=https://example.com); do \ - echo \"Site: \$site - Admin: \$(wp option get admin_email --path=web/wp --url=\$site)\"; \ - done" -``` - -#### Verify Email Address Exists - -Check if the admin email address is actually a working mailbox: -- Try sending a test email to the address from another account -- Check with your email hosting provider (Dreamhost, Google Workspace, etc.) -- Verify DNS MX records are correct for the domain - -#### Test WordPress Email Function - -Send a test email from WordPress: -```bash -ssh web@example.com "cd /srv/www/example.com/current && \ - wp eval \"wp_mail('your-real-email@example.com', 'Test Email', 'This is a test from WordPress'); \ - echo wp_mail('your-real-email@example.com', 'Test', 'Test') ? 'SUCCESS' : 'FAILED';\" \ - --path=web/wp --url=https://example.com" -``` - -### Solution - -#### Update Single Site Admin Email - -```bash -ssh web@example.com "cd /srv/www/example.com/current && \ - wp option update admin_email 'real-email@example.com' --path=web/wp" -``` - -#### Update Multisite Network Admin Email - -```bash -# Update network-wide admin email -ssh web@example.com "cd /srv/www/example.com/current && \ - wp site option update admin_email 'real-email@example.com' \ - --path=web/wp --url=https://example.com" - -# Update main site admin email -ssh web@example.com "cd /srv/www/example.com/current && \ - wp option update admin_email 'real-email@example.com' \ - --path=web/wp --url=https://example.com" -``` - -#### Update All Subsites (Bulk Operation) - -If you have multiple subsites with invalid admin emails: - -```bash -ssh web@example.com "cd /srv/www/example.com/current && \ - for site in \$(wp site list --field=url --path=web/wp --url=https://example.com); do \ - echo \"Updating \$site...\"; \ - wp option update admin_email 'real-email@example.com' --path=web/wp --url=\$site; \ - done" -``` - -#### Verify Changes - -```bash -# Check network admin email -ssh web@example.com "cd /srv/www/example.com/current && \ - wp site option get admin_email --path=web/wp --url=https://example.com" - -# Check all subsites -ssh web@example.com "cd /srv/www/example.com/current && \ - for site in \$(wp site list --field=url --path=web/wp --url=https://example.com); do \ - echo \"Site: \$site - Admin: \$(wp option get admin_email --path=web/wp --url=\$site)\"; \ - done" -``` - -### Testing - -#### Test Server-Level Email (sSMTP) - -Verify the server can send emails through SMTP relay: - -```bash -ssh web@example.com "echo -e 'Subject: Test Email\n\nThis is a test' | \ - /usr/sbin/sendmail -v your-email@example.com" -``` - -Expected output should show successful SMTP authentication and delivery: -``` -[<-] 220 smtp-relay.brevo.com ESMTP Service Ready -[->] EHLO example.com -[<-] 235 2.0.0 Authentication succeeded -[<-] 250 2.0.0 OK: queued as <...> -``` - -#### Test WordPress Email Function - -```bash -ssh web@example.com "cd /srv/www/example.com/current && \ - wp eval \" - \\\$to = 'your-email@example.com'; - \\\$subject = 'Test: WordPress Email'; - \\\$message = 'This is a test email from WordPress.'; - \\\$result = wp_mail(\\\$to, \\\$subject, \\\$message); - echo \\\$result ? 'Email sent successfully' : 'Email failed to send'; - \" --path=web/wp --url=https://example.com" -``` - -#### Test New Site Registration Email (Multisite) - -For multisite networks, create a test subsite to verify registration emails work: - -```bash -ssh web@example.com "cd /srv/www/example.com/current && \ - wp site create --slug=test-registration --title='Test Site' \ - --email=your-email@example.com --path=web/wp --url=https://example.com" -``` - -You should receive a "New Site Registration" email at your address. - -### Prevention - -#### Initial WordPress Setup - -When setting up a new WordPress site or multisite network: - -1. **Use a real, working email address** for the admin email during installation -2. **Verify the email address exists** before completing WordPress installation -3. **Test email delivery** immediately after setup - -#### Regular Audits - -Periodically check for invalid email addresses: - -```bash -# Audit all admin emails in multisite -ssh web@example.com "cd /srv/www/example.com/current && \ - for site in \$(wp site list --field=url --path=web/wp --url=https://example.com); do \ - email=\$(wp option get admin_email --path=web/wp --url=\$site); \ - echo \"Site: \$site\"; \ - echo \"Admin: \$email\"; \ - echo \"---\"; \ - done" -``` - -#### Email Monitoring - -- Use email service dashboards (Brevo, Mailgun, SendGrid) to monitor bounces -- Set up alerts for soft/hard bounces -- Regularly review email delivery reports - -#### Documentation - -Document the correct admin email addresses for: -- Production sites -- Staging sites -- Development/demo sites - -Keep this information in your project's `README.md` or operations documentation. - -#### Common Pitfalls to Avoid - -1. **Don't use non-existent subdomains**: Avoid `admin@demo.example.com` unless you've created that mailbox -2. **Don't use `.test` domains in production**: Replace development emails (`admin@example.test`) before deploying -3. **Don't assume email works**: Always test email delivery after WordPress setup or migrations -4. **Don't ignore bounce reports**: Soft bounces often indicate configuration issues \ No newline at end of file diff --git a/docs/troubleshooting/MariaDB.md b/docs/troubleshooting/MariaDB.md deleted file mode 100644 index b52e316..0000000 --- a/docs/troubleshooting/MariaDB.md +++ /dev/null @@ -1,362 +0,0 @@ -# MariaDB Troubleshooting Guide - -The provision failure walked through step by step: [Trellis Provision Fails at "Set root user password"](https://wpvilla.in/trellis-provision-set-root-user-password-fails-mariadb-provider-plugins/). - -This guide covers common MariaDB issues on Trellis-managed servers, including startup failures, plugin problems, and diagnostic approaches. - -## Table of Contents - -- [Common Errors](#common-errors) -- [Diagnostic Commands](#diagnostic-commands) -- [Compression Provider Plugin Failure](#compression-provider-plugin-failure) -- [Connection Issues](#connection-issues) -- [Performance Issues](#performance-issues) -- [Backup and Recovery](#backup-and-recovery) - -## Common Errors - -### Ansible "Set root user password" Fails - -**Symptom:** During Trellis provision, the MariaDB role fails with: -``` -TASK [mariadb : Set root user password] **************************************** -failed: [server-ip] (item=None) => {"censored": "...no_log: true..."} -``` - -**Common causes:** -1. MariaDB service is not running (most common) -2. Root password already set differently -3. Socket connection issues - -**First step:** Always check if MariaDB is actually running before assuming password issues. - -### Service Failed to Start - -**Symptom:** -``` -systemd[1]: mariadb.service: Main process exited, code=exited, status=7/NOTRUNNING -systemd[1]: mariadb.service: Failed with result 'exit-code'. -systemd[1]: Failed to start mariadb.service - MariaDB database server. -``` - -**Check the logs:** -```bash -sudo journalctl -u mariadb -n 50 -``` - -## Diagnostic Commands - -### Check Service Status - -```bash -# Service status -sudo systemctl status mariadb - -# Is MariaDB process running? -sudo ps aux | grep mysql - -# Recent logs -sudo journalctl -u mariadb -n 50 - -# Watch logs in real-time -sudo journalctl -u mariadb -f -``` - -### Check Database Connectivity - -```bash -# Test connection (get password from vault) -mysql -u root -p'[password]' -e "SELECT 1;" - -# Check WordPress database -mysql -u root -p'[password]' -e "SHOW DATABASES;" - -# Check user privileges -mysql -u root -p'[password]' -e "SELECT user, host FROM mysql.user;" -``` - -### Check Configuration - -```bash -# Main configuration -cat /etc/mysql/mariadb.conf.d/50-server.cnf - -# All configuration files -ls -la /etc/mysql/mariadb.conf.d/ - -# Search for specific settings -grep -r "innodb" /etc/mysql/ -``` - -### Check Resources - -```bash -# Disk space (MariaDB needs space for logs, temp tables) -df -h - -# Memory usage -free -h - -# MariaDB memory consumption -ps aux | grep mariadbd | awk '{print $6/1024" MB"}' -``` - -## Compression Provider Plugin Failure - -A common issue on Ubuntu 24.04 with MariaDB 10.11+. - -### Symptoms - -MariaDB fails to start with errors like: -``` -[ERROR] mariadbd: Can't open shared library '/usr/lib/mysql/plugin/provider_bzip2.so' -[ERROR] Couldn't load plugins from 'provider_bzip2.so'. -[ERROR] /usr/sbin/mariadbd: unknown variable 'provider_bzip2=force_plus_permanent' -[ERROR] Aborting -``` - -### Cause - -Configuration files exist for compression plugins that aren't installed: -- `/etc/mysql/mariadb.conf.d/provider_bzip2.cnf` -- `/etc/mysql/mariadb.conf.d/provider_lz4.cnf` -- `/etc/mysql/mariadb.conf.d/provider_lzma.cnf` -- `/etc/mysql/mariadb.conf.d/provider_lzo.cnf` -- `/etc/mysql/mariadb.conf.d/provider_snappy.cnf` - -This happens when: -- MariaDB was upgraded without installing plugin packages -- Ubuntu repository restructured plugin distribution -- Config files were manually added but packages never installed - -### Solution A: Disable Plugins (Recommended) - -These plugins are optional and not needed for WordPress: - -```bash -# Disable all compression provider configs -sudo mv /etc/mysql/mariadb.conf.d/provider_lz4.cnf /etc/mysql/mariadb.conf.d/provider_lz4.cnf.disabled -sudo mv /etc/mysql/mariadb.conf.d/provider_snappy.cnf /etc/mysql/mariadb.conf.d/provider_snappy.cnf.disabled -sudo mv /etc/mysql/mariadb.conf.d/provider_lzo.cnf /etc/mysql/mariadb.conf.d/provider_lzo.cnf.disabled -sudo mv /etc/mysql/mariadb.conf.d/provider_bzip2.cnf /etc/mysql/mariadb.conf.d/provider_bzip2.cnf.disabled -sudo mv /etc/mysql/mariadb.conf.d/provider_lzma.cnf /etc/mysql/mariadb.conf.d/provider_lzma.cnf.disabled - -# Start MariaDB -sudo systemctl start mariadb -sudo systemctl status mariadb -``` - -### Solution B: Install Plugin Packages - -If you need compression plugins: - -```bash -sudo apt update -sudo apt install mariadb-plugin-provider-bzip2 \ - mariadb-plugin-provider-lz4 \ - mariadb-plugin-provider-lzma \ - mariadb-plugin-provider-lzo \ - mariadb-plugin-provider-snappy - -sudo systemctl restart mariadb -``` - -**Note:** Package names may vary by MariaDB version and Ubuntu release. - -### Impact of Disabling Compression Plugins - -**Low impact for WordPress sites:** -- WordPress databases don't use table-level compression by default -- Default zlib compression remains available (built-in) -- No data loss or performance degradation -- Only affects tables explicitly using `ROW_FORMAT=COMPRESSED` - -## Connection Issues - -### Socket Connection Errors - -**Symptom:** -``` -ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' -``` - -**Solutions:** - -```bash -# Check if socket exists -ls -la /var/run/mysqld/ - -# Check socket configuration -grep socket /etc/mysql/mariadb.conf.d/50-server.cnf - -# Restart MariaDB -sudo systemctl restart mariadb -``` - -### Access Denied Errors - -**Symptom:** -``` -ERROR 1045 (28000): Access denied for user 'root'@'localhost' -``` - -**Check credentials in Trellis vault:** -```bash -# View vault (in your trellis directory) -ansible-vault view group_vars/production/vault.yml -``` - -**Reset root password (emergency):** -```bash -sudo systemctl stop mariadb -sudo mysqld_safe --skip-grant-tables & -mysql -u root - -# In MySQL shell: -FLUSH PRIVILEGES; -ALTER USER 'root'@'localhost' IDENTIFIED BY 'new_password'; -FLUSH PRIVILEGES; -EXIT; - -sudo systemctl restart mariadb -``` - -## Performance Issues - -### Slow Queries - -**Enable slow query log:** - -Add to `/etc/mysql/mariadb.conf.d/50-server.cnf`: -```ini -slow_query_log = 1 -slow_query_log_file = /var/log/mysql/slow.log -long_query_time = 2 -``` - -**Analyze slow queries:** -```bash -sudo tail -50 /var/log/mysql/slow.log -``` - -### High Memory Usage - -**Check current settings:** -```bash -mysql -u root -p -e "SHOW VARIABLES LIKE 'innodb_buffer_pool_size';" -mysql -u root -p -e "SHOW VARIABLES LIKE 'key_buffer_size';" -``` - -**Recommended settings for small servers (4GB RAM):** -```ini -# In /etc/mysql/mariadb.conf.d/50-server.cnf -innodb_buffer_pool_size = 512M -key_buffer_size = 32M -``` - -### Table Lock Contention - -**Check for locks:** -```bash -mysql -u root -p -e "SHOW PROCESSLIST;" -mysql -u root -p -e "SHOW ENGINE INNODB STATUS\G" | grep -A 20 "TRANSACTIONS" -``` - -## Backup and Recovery - -### Quick Database Backup - -```bash -# Backup single database -mysqldump -u root -p'[password]' database_name > backup.sql - -# Backup all databases -mysqldump -u root -p'[password]' --all-databases > all_databases.sql - -# Compressed backup -mysqldump -u root -p'[password]' database_name | gzip > backup.sql.gz -``` - -### Restore Database - -```bash -# Restore from SQL file -mysql -u root -p'[password]' database_name < backup.sql - -# Restore from compressed backup -gunzip < backup.sql.gz | mysql -u root -p'[password]' database_name -``` - -### Check Database Integrity - -```bash -# Check all tables in a database -mysqlcheck -u root -p'[password]' --check database_name - -# Repair tables -mysqlcheck -u root -p'[password]' --repair database_name -``` - -## After Fixing Issues - -### Verify MariaDB is Running - -```bash -sudo systemctl status mariadb -# Should show: Active: active (running) -``` - -### Test Database Connection - -```bash -mysql -u root -p'[password]' -e "SELECT 1;" -``` - -### Continue Trellis Provision - -```bash -cd trellis -trellis provision --tags mariadb production -# Or full provision -trellis provision production -``` - -### Test WordPress Sites - -After fixing MariaDB issues, verify your WordPress sites are accessible and database operations work correctly. - -## Prevention Tips - -1. **Test provisions on staging first** - Always test major infrastructure changes before production - -2. **Monitor MariaDB logs** - Set up log monitoring: - ```bash - sudo journalctl -u mariadb -f - ``` - -3. **Regular backups** - Use the backup playbooks in this repository - -4. **Check disk space** - MariaDB needs space for logs and temporary tables: - ```bash - df -h /var/lib/mysql - ``` - -5. **Review Trellis updates** - When updating Trellis, review MariaDB role changes - -## Lessons Learned - -1. **Check service status first** - When database tasks fail, verify the service is running before assuming authentication issues - -2. **Misleading Ansible errors** - "Set root user password failed" often means MariaDB isn't running, not a password problem - -3. **Review full logs** - `journalctl -u mariadb` reveals the actual problem, which Ansible output may hide - -4. **Optional plugins can break core service** - Missing plugin libraries can prevent MariaDB from starting entirely - -5. **Simple solutions work** - Disabling unused optional features is often better than debugging them - -## Related Resources - -- [MariaDB Documentation](https://mariadb.com/kb/en/) -- [Trellis MariaDB Configuration](https://roots.io/trellis/docs/database-access/) -- [MariaDB Plugin Overview](https://mariadb.com/kb/en/plugin-overview/) diff --git a/docs/troubleshooting/OOM.md b/docs/troubleshooting/OOM.md deleted file mode 100644 index 1e979e0..0000000 --- a/docs/troubleshooting/OOM.md +++ /dev/null @@ -1,346 +0,0 @@ -# Out of Memory (OOM) Troubleshooting Guide - -## Table of Contents - -1. [Problem Overview](#problem-overview) -2. [Quick Diagnosis Commands](#quick-diagnosis-commands) -3. [The Discovery](#the-discovery) -4. [Root Cause Analysis](#root-cause-analysis) -5. [Investigation Process](#investigation-process) - - [System Cron Jobs Audit](#system-cron-jobs-audit) - - [WordPress Cron Events](#wordpress-cron-events) - - [Action Scheduler Context](#action-scheduler-context) - - [System Status Check](#system-status-check) -6. [Trellis WP-Cron Configuration](#trellis-wp-cron-configuration) -7. [Next Steps](#next-steps) - -## Problem Overview - -When a server runs out of memory, the Linux kernel's OOM (Out of Memory) killer terminates processes to free up resources. This guide documents troubleshooting an OOM issue on a 4GB Trellis/WordPress server where processes were being killed due to memory exhaustion. - -## Quick Diagnosis Commands - -### Check for OOM Events in Previous Boot - -The most effective command to find OOM killer events: - -```sh -ssh root@domain.com 'journalctl -k -b -2 | grep -Ei "oom|killed process|out of memory"' -``` - -This checks the kernel logs from 2 boots ago (`-b -2`), which is crucial if the server has been restarted since the issue occurred. - -### Check Current Memory Usage - -```sh -ssh root@domain.com "ps aux --sort=-%mem | head -10" -``` - -### Check PHP-FPM Worker Memory - -```sh -ssh root@domain.com "ps aux --sort=-%mem | grep 'php-fpm: pool' | head -5 | awk '{print \$6/1024 \" MB - PID \" \$2}'" -``` - -### Check WordPress Cron Events - -```sh -ssh web@domain.com "cd /srv/www/domain.com/current && wp cron event list --path=web/wp" -``` - -### Audit All System Cron Jobs - -```sh -ssh root@domain.com 'ls -lah /etc/cron* /var/spool/cron/crontabs' -``` - -## The Discovery - -After checking current boot logs and finding no issues, the critical discovery came from checking **previous boot logs**: - -```sh -ssh root@domain.com 'journalctl -k -b -2 | grep -Ei "oom|killed process|out of memory"' -``` - -### The Smoking Gun - -Two OOM events were found on November 24, approximately 30 minutes apart: - -**Event 1 - Nov 24 01:48:27:** -``` -Nov 24 01:48:27 server kernel: systemd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 -Nov 24 01:48:27 server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/cron.service,task=php,pid=70787,uid=1000 -Nov 24 01:48:27 server kernel: Out of memory: Killed process 70787 (php) total-vm:3621976kB, anon-rss:3235148kB, file-rss:5760kB, shmem-rss:69120kB, UID:1000 pgtables:6960kB oom_score_adj:0 -``` - -**Event 2 - Nov 24 02:21:02:** -``` -Nov 24 02:21:02 server kernel: systemd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 -Nov 24 02:21:02 server kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/cron.service,task=php,pid=71246,uid=1000 -Nov 24 02:21:02 server kernel: Out of memory: Killed process 71246 (php) total-vm:3236952kB, anon-rss:2517056kB, file-rss:5632kB, shmem-rss:69120kB, UID:1000 pgtables:6188kB oom_score_adj:0 -``` - -## Root Cause Analysis - -### Key Findings from OOM Logs - -Both OOM events show identical characteristics: - -1. **Process Name:** `php` (NOT `php-fpm: pool wordpress`) -2. **User ID:** `1000` (deploy user, not `www-data`) -3. **Parent Service:** `task_memcg=/system.slice/cron.service` -4. **Memory Consumption:** 2.5-3.2 GB RSS per process - - Event 1: `anon-rss:3235148kB` (3.08 GB) - - Event 2: `anon-rss:2517056kB` (2.40 GB) - -### Critical Insight - -🔥 **This is NOT a PHP-FPM issue.** - -The OOM logs clearly show: -- **Standalone PHP processes** triggered by system cron -- Each consuming 2.5-3.2 GB of RAM -- On a 4GB server, this is catastrophic - -The key indicator: -``` -task_memcg=/system.slice/cron.service, task=php, uid=1000 -``` - -This means: -- ❗ A **cron job** is running a PHP script -- ❗ That script consumes massive memory (likely a leak or heavy task) -- ❗ PHP-FPM workers were healthy (~120-130 MB each) during the incident -- ❗ The culprit runs every ~30 minutes based on timing - -## Investigation Process - -### System Cron Jobs Audit - -Checking all cron jobs on the system: - -```sh -ssh root@domain.com 'ls -lah /etc/cron* /var/spool/cron/crontabs' -``` - -**Relevant cron files in `/etc/cron.d/`:** - -``` --rw-r--r-- 1 root root 712 Sep 27 2024 php --rw-r--r-- 1 root root 162 Nov 22 04:28 php-fpm-monitor --rw-r--r-- 1 root root 138 Feb 13 2025 wordpress-example_com --rw-r--r-- 1 root root 210 Apr 4 2025 wordpress-multisite-example_com --rw-r--r-- 1 root root 141 Feb 13 2025 letsencrypt-certificate-renewal -``` - -**WordPress cron jobs** are the prime suspects: -- `wordpress-example_com` - Main site WP-Cron (every 15 minutes) -- `wordpress-multisite-example_com` - Multisite WP-Cron -- These trigger curl requests to wp-cron.php which spawns PHP processes - -### WordPress Cron Events - -Current scheduled WordPress cron events: - -```sh -ssh web@domain.com "cd /srv/www/domain.com/current && wp cron event list --path=web/wp" -``` - -**Output:** -``` -hook next_run_gmt next_run_relative recurrence -action_scheduler_run_queue 2025-11-25 03:45:57 now 1 minute -rocket_update_dynamic_lists 2025-11-25 03:56:19 now 1 week -rocket_preload_clean_rows_time_event 2025-11-25 04:06:19 8 minutes 27 seconds 1 week -jetpack_clean_nonces 2025-11-25 04:20:32 22 minutes 40 seconds 1 hour -wp_privacy_delete_old_export_files 2025-11-25 04:33:15 35 minutes 23 seconds 1 hour -wc_admin_process_orders_milestone 2025-11-25 04:40:31 42 minutes 39 seconds 1 hour -wc_admin_unsnooze_admin_notes 2025-11-25 04:42:46 44 minutes 54 seconds 1 hour -wp_scheduled_delete 2025-11-25 05:13:18 1 hour 15 minutes 1 day -delete_expired_transients 2025-11-25 05:13:18 1 hour 15 minutes 1 day -``` - -**Suspects:** -- `action_scheduler_run_queue` - Runs every minute, processes WooCommerce background jobs -- `rocket_update_dynamic_lists` - WP Rocket cache plugin -- `wc_admin_*` - WooCommerce admin tasks - -### Action Scheduler Context - -#### Previous Mitigation Attempt - -A MU plugin was previously deployed to disable Action Scheduler's async runner: - -`app/mu-plugins/disable-action-scheduler-async.php`: -```php - /dev/null 2>&1" - cron_file: "wordpress-{{ item.key | replace('.', '_') }}" - with_dict: "{{ wordpress_sites }}" - when: site_env.disable_wp_cron and not item.value.multisite.enabled | default(false) -``` - -This creates `/etc/cron.d/wordpress-domain_com` that runs every 15 minutes. - -**The Problem:** -- Cron triggers `wp-cron.php` via curl -- WordPress spawns a **standalone PHP process** (not PHP-FPM) -- This process uses **PHP CLI settings**, not PHP-FPM pool limits -- Trellis provides `php_cli_memory_limit` variable (default: `-1` unlimited) -- **If not explicitly set** in `group_vars/*/main.yml`, PHP CLI has unlimited memory -- Heavy scheduled tasks can consume unbounded memory until OOM -- On 4GB server, 3GB consumption = catastrophic - -## Next Steps - -### Immediate Actions - -1. **Verify PHP CLI memory limit is unlimited:** - ```sh - ssh root@domain.com "php -i | grep memory_limit" - ``` - - **Expected output:** - ``` - memory_limit => -1 => -1 - ``` - - This confirms PHP CLI has **unlimited memory**, which is the root cause. - - **Fix via Trellis:** Add `php_cli_memory_limit` to your `group_vars/production/main.yml`: - ```yaml - php_cli_memory_limit: 512M - ``` - - Then re-provision: - ```sh - trellis provision production - ``` - - **Manual fix (alternative):** Edit `/etc/php/8.3/cli/php.ini` directly: - ```ini - memory_limit = 512M - ``` - - **Note:** Trellis provides the `php_cli_memory_limit` variable (from `roles/php/defaults/main.yml`), but it defaults to `-1` (unlimited) if not explicitly set in your `group_vars/*/main.yml`. - -2. **Identify the problematic WordPress cron task:** - - Check Action Scheduler queue for stuck/failed jobs - - Review WP Rocket cache tasks - - Check WooCommerce admin tasks - - Monitor which cron hook runs when OOM occurs - -3. **Add monitoring:** - ```sh - # Log PHP processes spawned by cron - */5 * * * * root ps aux | grep -E "php.*cron" >> /var/log/php-cron-monitor.log - ``` - -4. **Consider memory limits per cron job:** - Modify WP-Cron trigger to include memory limit: - ```sh - php -d memory_limit=512M wp-cron.php - ``` - -### Long-term Solutions - -1. **Upgrade server RAM** if workload requires it (consider 8GB) -2. **Optimize heavy scheduled tasks:** - - Break large tasks into smaller batches - - Add time/memory limits to custom cron jobs - - Review Action Scheduler queue regularly -3. **Implement proper monitoring:** - - Alert on OOM events - - Track PHP process memory consumption - - Monitor cron job execution times -4. **Consider dedicated cron server** for large sites - -### Verification - -After implementing fixes, verify with: -```sh -# Check for new OOM events (run daily) -ssh root@domain.com 'journalctl -k --since "24 hours ago" | grep -Ei "oom|killed process"' - -# Monitor PHP CLI process memory -ssh root@domain.com 'watch "ps aux | grep php | grep -v php-fpm | grep -v grep"' -``` diff --git a/docs/troubleshooting/PHP-FPM.md b/docs/troubleshooting/PHP-FPM.md deleted file mode 100644 index f98a477..0000000 --- a/docs/troubleshooting/PHP-FPM.md +++ /dev/null @@ -1,414 +0,0 @@ -# PHP-FPM Troubleshooting Guide - -A worked case study using this guide: [Debugging PHP-FPM Memory Exhaustion on WordPress with WooCommerce and Trellis](https://wpvilla.in/debugging-php-fpm-memory-exhaustion-wordpress-woocommerce-trellis/). - -This guide covers PHP-FPM pool management, memory configuration, and troubleshooting for Trellis-managed WordPress servers. - -## Table of Contents - -- [Common Errors](#common-errors) -- [Understanding PHP-FPM Dynamic Pools](#understanding-php-fpm-dynamic-pools) -- [Memory Math: Calculating Safe Worker Counts](#memory-math-calculating-safe-worker-counts) -- [WordPress Memory Limits](#wordpress-memory-limits) -- [Worker Memory Accumulation](#worker-memory-accumulation) -- [The Low-Traffic Recycling Problem](#the-low-traffic-recycling-problem) -- [Recommended Configurations](#recommended-configurations) -- [Diagnostic Commands](#diagnostic-commands) -- [Applying Changes via Trellis](#applying-changes-via-trellis) -- [Solutions by Problem](#solutions-by-problem) - -## Common Errors - -### Pool Exhaustion - -**Log message:** -``` -[pool wordpress] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 22 total children -[pool wordpress] server reached pm.max_children setting (30), consider raising it -``` - -**Symptom:** WordPress displays "There has been a critical error on this website" - -**What's happening:** -1. All PHP-FPM workers are busy processing requests -2. New request comes in -3. No available workers → request fails -4. WordPress returns generic "critical error" message - -### Memory Exhaustion - -**Log message:** -``` -Fatal error: Allowed memory size of 41943040 bytes exhausted -``` - -**What's happening:** WordPress's internal memory limit (often 40M default) is too low for plugins like WooCommerce or Acorn/Laravel. - -## Understanding PHP-FPM Dynamic Pools - -Trellis uses `pm = dynamic` by default, which spawns workers based on demand. - -| Setting | Purpose | -|---------|---------| -| `pm.max_children` | Maximum workers allowed | -| `pm.start_servers` | Workers spawned on startup | -| `pm.min_spare_servers` | Minimum idle workers to maintain | -| `pm.max_spare_servers` | Maximum idle workers before killing extras | -| `pm.max_requests` | Requests per worker before recycling | - -**Default Trellis values:** -```yaml -php_fpm_pm_max_children: 10 -php_fpm_pm_start_servers: 2 -php_fpm_pm_min_spare_servers: 1 -php_fpm_pm_max_spare_servers: 3 -php_fpm_pm_max_requests: 500 -``` - -## Memory Math: Calculating Safe Worker Counts - -Each PHP-FPM worker typically uses 100-250MB depending on your plugins. - -### Formula - -``` -Safe max_children = Available RAM / Average worker memory -``` - -### Example Calculations - -**4GB Server:** -``` -Available RAM: ~3.5GB (after system/services) -Average worker: ~200MB -Safe workers: 3500MB / 200MB = ~17 workers -``` - -**8GB Server:** -``` -Available RAM: ~7GB (after system/services) -Average worker: ~200MB -Safe workers: 7000MB / 200MB = ~35 workers -``` - -### The Resource Ceiling Conflict - -On smaller servers, you face a trade-off: -- **For memory safety:** Fewer workers (prevent OOM) -- **For traffic handling:** More workers (handle concurrent requests) - -**Solutions:** -1. Upgrade server RAM -2. Reduce worker memory via `pm.max_requests` -3. Add Redis caching to reduce per-request load -4. Optimize WordPress (fewer plugins, cleaner autoload) - -## WordPress Memory Limits - -WordPress has its **own memory limits** separate from PHP's limit. - -| Setting | Purpose | WordPress Default | -|---------|---------|-------------------| -| `WP_MEMORY_LIMIT` | Memory for frontend requests | 40M | -| `WP_MAX_MEMORY_LIMIT` | Memory for admin/backend | 256M | -| `php_memory_limit` | PHP-FPM's overall limit | 512M (Trellis) | - -**Key insight:** Even if PHP allows 768MB, WordPress caps itself at `WP_MEMORY_LIMIT` (40MB by default). This is often the hidden cause of memory errors. - -### Setting WordPress Memory Limits - -In your Bedrock site's `config/application.php`: - -```php -/** - * Memory Limits - * - * Increased for WooCommerce + modern themes/plugins. - */ -Config::define('WP_MEMORY_LIMIT', '256M'); -Config::define('WP_MAX_MEMORY_LIMIT', '512M'); -``` - -### Verify Memory Limits - -```bash -wp eval 'echo "WP_MEMORY_LIMIT: " . WP_MEMORY_LIMIT . "\nWP_MAX_MEMORY_LIMIT: " . WP_MAX_MEMORY_LIMIT;' --path=web/wp -``` - -## Worker Memory Accumulation - -PHP-FPM workers don't release memory between requests. Over time, they accumulate: - -``` -Request 1: 150 MB -Request 50: 250 MB -Request 100: 350 MB -Request 200: 500+ MB ← Danger zone! -``` - -### The Solution: Worker Recycling - -`pm.max_requests` controls how many requests a worker handles before being killed and respawned: - -```yaml -pm.max_requests = 100 # Kill worker after 100 requests -``` - -**Trade-off:** -- Lower value = More recycling overhead, safer memory -- Higher value = Less overhead, risk of memory bloat - -**Recommended:** Start with `100` for memory-heavy sites (WooCommerce, Acorn), use `200-500` for lighter sites. - -### Testing Worker Recycling - -```bash -# Generate traffic -for i in {1..200}; do - curl -s -o /dev/null "https://example.com/?nocache=$(date +%s%N)$i" & -done -wait - -# Check worker memory (should be low after recycling) -ps aux --sort=-%mem | grep 'php-fpm: pool' | head -5 | awk '{print $6/1024" MB - PID "$2}' -``` - -**Expected:** Workers should be 130-160MB after recycling, not 400MB+. - -### The Low-Traffic Recycling Problem - -On low-traffic sites, `pm.max_requests` may be ineffective because workers don't receive enough requests to trigger recycling: - -``` -Traffic rate: ~80 requests/hour -Workers: 15 -Requests per worker per hour: ~5 -Time to reach 100 requests: ~20 hours - -Meanwhile, memory accumulates faster than recycling occurs. -``` - -**Symptoms:** -- Workers showing 500MB-800MB+ despite `pm.max_requests: 100` -- Memory exhaustion errors on basic requests (feeds, robots.txt, monitoring pings) -- Errors from uptime monitors and crawlers - -**Solutions for low-traffic sites:** - -| Option | Description | Trade-off | -|--------|-------------|-----------| -| Reduce `pm.max_requests` to 25-50 | More aggressive recycling | More spawn overhead | -| Reduce `pm.min_spare_servers` | Fewer idle workers accumulating memory | Slower response to traffic spikes | -| Add `pm.process_idle_timeout` | Kill idle workers after X seconds | May slow response times | -| Scheduled PHP-FPM restart | Cron job to restart nightly | Brief service interruption | -| Enable Redis object cache | Reduce per-request memory usage | Setup complexity | - -**Recommended for low-traffic sites:** - -```yaml -# Fewer workers, more aggressive recycling -php_fpm_pm_max_children: 15 -php_fpm_pm_start_servers: 4 -php_fpm_pm_min_spare_servers: 2 -php_fpm_pm_max_spare_servers: 6 -php_fpm_pm_max_requests: 50 -``` - -## Recommended Configurations - -### 4GB Server (Conservative) - -```yaml -# trellis/group_vars/production/main.yml -php_fpm_pm: dynamic -php_fpm_pm_max_children: 20 -php_fpm_pm_start_servers: 8 -php_fpm_pm_min_spare_servers: 6 -php_fpm_pm_max_spare_servers: 12 -php_fpm_pm_max_requests: 100 - -php_memory_limit: 768M -``` - -### 4GB Server (Aggressive) - -For sites with traffic spikes, accepting some memory risk: - -```yaml -php_fpm_pm_max_children: 25 -php_fpm_pm_start_servers: 12 -php_fpm_pm_min_spare_servers: 10 -php_fpm_pm_max_spare_servers: 18 -php_fpm_pm_max_requests: 100 -``` - -### 8GB Server - -```yaml -php_fpm_pm_max_children: 40 -php_fpm_pm_start_servers: 15 -php_fpm_pm_min_spare_servers: 10 -php_fpm_pm_max_spare_servers: 25 -php_fpm_pm_max_requests: 200 - -php_memory_limit: 768M -``` - -### High-Traffic / WooCommerce - -```yaml -php_fpm_pm_max_children: 50 -php_fpm_pm_start_servers: 20 -php_fpm_pm_min_spare_servers: 15 -php_fpm_pm_max_spare_servers: 30 -php_fpm_pm_max_requests: 100 # Keep low for WooCommerce -``` - -## Diagnostic Commands - -### Check Worker Status - -```bash -# Count active workers -pgrep -c php-fpm - -# Memory per worker (replace 8.3 with your PHP version) -ps aux --sort=-%mem | grep 'php-fpm: pool' | head -10 | awk '{print $6/1024" MB - PID "$2}' - -# Total PHP-FPM memory -ps aux | grep 'php-fpm: pool' | awk '{sum+=$6} END {print sum/1024" MB"}' - -# Current pool configuration -cat /etc/php/8.3/fpm/pool.d/wordpress.conf | grep -E '^pm' -``` - -### Check Logs - -```bash -# Pool exhaustion warnings -grep 'seems busy\|max_children' /var/log/php8.3-fpm.log | tail -20 - -# Watch logs in real-time -tail -f /var/log/php8.3-fpm.log -``` - -### Check System Memory - -```bash -# Memory overview -free -h - -# Detailed memory info -cat /proc/meminfo | grep -E 'MemTotal|MemAvailable|SwapTotal|SwapFree' - -# Top memory consumers -ps aux --sort=-%mem | head -15 -``` - -### Check Autoload Options - -Large autoloaded options increase per-request memory: - -```bash -# Count and size -wp db query "SELECT COUNT(*) as count, ROUND(SUM(LENGTH(option_value))/1024) as size_kb FROM wp_options WHERE autoload='yes';" --path=web/wp - -# Largest options -wp db query "SELECT option_name, LENGTH(option_value) as size_bytes FROM wp_options WHERE autoload='yes' ORDER BY size_bytes DESC LIMIT 10;" --path=web/wp -``` - -### Quick Health Check - -Comprehensive one-liner to check PHP-FPM configuration and status: - -```bash -ssh root@your-server.com "echo '=== PHP-FPM Pool Config ===' && cat /etc/php/8.3/fpm/pool.d/wordpress.conf | grep -E '^pm' && echo '' && echo '=== PHP Memory Limit ===' && grep memory_limit /etc/php/8.3/fpm/php.ini && echo '' && echo '=== Current Workers ===' && ps aux --sort=-%mem | grep 'php-fpm: pool' | head -5 | awk '{print \$6/1024 \" MB - PID \" \$2}' && echo '' && echo '=== Memory ===' && free -m | grep Mem" -``` - -**Note:** Replace `8.3` with your PHP version. - -**Healthy output:** -- Pool config: Shows your pm.* settings -- Memory limit: Should match your Trellis config -- Workers: Each worker < 200MB -- Memory: Available RAM > 1GB - -## Applying Changes via Trellis - -### Update Configuration - -Edit `trellis/group_vars/production/main.yml`: - -```yaml -php_fpm_pm: dynamic -php_fpm_pm_max_children: 25 -php_fpm_pm_start_servers: 10 -php_fpm_pm_min_spare_servers: 8 -php_fpm_pm_max_spare_servers: 15 -php_fpm_pm_max_requests: 100 -``` - -### Apply Changes - -```bash -cd trellis -trellis provision --tags wordpress-setup production -``` - -This updates `/etc/php/8.3/fpm/pool.d/wordpress.conf` and reloads PHP-FPM. - -### Verify Changes - -```bash -ssh root@your-server.com "cat /etc/php/8.3/fpm/pool.d/wordpress.conf | grep -E '^pm'" -``` - -### Manual PHP-FPM Restart - -If provisioning doesn't restart PHP-FPM: - -```bash -ssh root@your-server.com "systemctl restart php8.3-fpm" -``` - -## Solutions by Problem - -### Problem: "Critical error on this website" - -1. Check `/var/log/php8.3-fpm.log` for `max_children` warnings -2. Check worker memory with `ps aux` -3. If workers > 200MB each: reduce `pm.max_requests` to 100 -4. If pool exhausted: increase `max_children` (if RAM allows) or upgrade server - -### Problem: Memory exhaustion errors - -1. Check WordPress limits: `wp eval 'echo WP_MEMORY_LIMIT;'` -2. If 40M: add `WP_MEMORY_LIMIT` to `application.php` -3. Check `php_memory_limit` in Trellis config -4. Check for bloated transients in database - -### Problem: Slow admin panel - -1. Check autoloaded options size (should be < 1MB) -2. Consider Redis caching -3. Clean up expired transients: `wp transient delete --expired` - -### Problem: 504 Gateway Timeout - -1. Usually indicates workers are all busy -2. Check `max_children` setting -3. Increase idle workers (`min_spare_servers`, `start_servers`) -4. Look for long-running requests in access logs - -### Problem: OOM (Out of Memory) during deploy - -1. Reduce `max_children` temporarily -2. Deploy during low-traffic period -3. Upgrade server RAM -4. Check if `wp acorn optimize` is running (needs ~3GB headroom) - -## Additional Resources - -- [Trellis PHP Configuration](https://roots.io/trellis/docs/php/) -- [Trellis Redis Setup](https://roots.io/trellis/docs/redis/) -- [PHP-FPM Configuration Reference](https://www.php.net/manual/en/install.fpm.configuration.php) diff --git a/docs/troubleshooting/README.md b/docs/troubleshooting/README.md deleted file mode 100644 index f0345ad..0000000 --- a/docs/troubleshooting/README.md +++ /dev/null @@ -1,87 +0,0 @@ -# Troubleshooting WordPress Servers - -This section contains guides for diagnosing and resolving common issues with WordPress servers. While examples reference Trellis-managed servers, the diagnostic techniques and solutions apply to most WordPress hosting environments. - -## Guides - -| Guide | Description | -|-------|-------------| -| [PHP-FPM.md](PHP-FPM.md) | Pool exhaustion, memory management, worker configuration | -| [MariaDB.md](MariaDB.md) | Startup failures, plugin issues, connection problems | -| [OOM.md](OOM.md) | Out of memory issues, WP-Cron memory leaks, PHP CLI limits | -| [MAIL.md](MAIL.md) | SMTP configuration issues after Trellis upgrades | -| [Multisite subsite rename](https://wpvilla.in/wordpress-multisite-rename-subsite-slug-stale-cache-wp-cli/) | Blog post: renamed subsite still shows the old slug in WP-CLI | - -## Quick Diagnostic Commands - -SSH to your server and run these commands to quickly assess server health. - -### System Memory - -```bash -# Memory overview -free -h - -# Top memory consumers -ps aux --sort=-%mem | head -10 -``` - -### PHP-FPM Workers - -```bash -# Count active workers -pgrep -c php-fpm - -# Memory usage per worker (replace 8.3 with your PHP version) -ps aux --sort=-%mem | grep 'php-fpm: pool' | head -5 | awk '{print $6/1024" MB - PID "$2}' - -# Total PHP-FPM memory usage -ps aux | grep 'php-fpm: pool' | awk '{sum+=$6} END {print sum/1024" MB"}' - -# Current pool configuration -cat /etc/php/8.3/fpm/pool.d/wordpress.conf | grep -E '^pm' -``` - -### PHP-FPM Logs - -```bash -# Recent warnings (replace 8.3 with your PHP version) -grep 'seems busy\|max_children' /var/log/php8.3-fpm.log | tail -20 - -# Watch logs in real-time -tail -f /var/log/php8.3-fpm.log -``` - -### Nginx Logs - -```bash -# Recent errors -tail -50 /var/log/nginx/error.log - -# Site-specific errors -tail -50 /srv/www/example.com/logs/error.log -``` - -### WordPress Diagnostics - -```bash -# Check WordPress memory limits (run as web user) -cd /srv/www/example.com/current -wp eval 'echo "WP_MEMORY_LIMIT: " . WP_MEMORY_LIMIT . "\nWP_MAX_MEMORY_LIMIT: " . WP_MAX_MEMORY_LIMIT;' --path=web/wp - -# Check autoloaded options size -wp db query "SELECT COUNT(*) as count, ROUND(SUM(LENGTH(option_value))/1024) as size_kb FROM wp_options WHERE autoload='yes';" --path=web/wp -``` - -## Quick Health Check - -One-liner to check overall server health: - -```bash -ssh root@your-server.com "echo '=== Memory ===' && free -m | grep Mem && echo '' && echo '=== PHP-FPM Workers ===' && ps aux --sort=-%mem | grep 'php-fpm: pool' | head -5 | awk '{print \$6/1024 \" MB - PID \" \$2}'" -``` - -**Healthy indicators:** -- Memory: Available RAM > 1GB -- Workers: Each worker < 200MB -- No recent errors in logs diff --git a/docs/wordpress-utilities/README.md b/docs/wordpress-utilities/README.md index 625ac17..5f009af 100644 --- a/docs/wordpress-utilities/README.md +++ b/docs/wordpress-utilities/README.md @@ -153,7 +153,6 @@ These utilities are provided as-is for use in WordPress projects. Refer to indiv For issues or questions: - Review the specific utility's README.md - Check WordPress Codex and plugin documentation -- Consult related troubleshooting guides in [../troubleshooting/](../troubleshooting/) ---