Revisit disabling Indexables via wp_get_environment_type() and provide different solution (provided)

[mkormendy]

The Problem

Discussed in length in the following issues:
#22030
#21459

Essentially, the PR solution 21335:

• contains a patchwork of poorly handled workarounds for URL/domain-name differences in the SEO optimized data that is migrated between staging and production environments
• broke the automatic indexables optimizations on non-production environments where those benefits were utilized and accounted for in properly configured migration strategies

This problem was introduced in PR 21335. Staging environments set with the corresponding environment variable in the wp-config file now have stale meta data and non-indexed content in the indexables tables. The suggested solution now forces the indexables optimization to be turned off on all environments that are not production, and further unnecessary manual steps are required to maintain the indexables approach that was previously automatic. Worse, turning off indexables altogether with the introduction of a filter hook also has negative effects for breadcrumbs, opengraph and twitter meta images as warned in the the Yoast support documentation.

Proposed Solutions

There are two different succinct elegant approaches to resolving both issues above entirely.

Solution One (more straightforward but requires user interaction)

• Rollback PR 21335 and don't use a common environment variable (avoiding implications for other operations outside of Yoast's operation).
• Simply provide an option in the Yoast SEO plugin settings to turn indexables off or on for that installation.
• Move the reset and reindexing feature from the Yoast Test Helper plugin to the main Yoast SEO plugin underneath the Tools menu.

Solution Two (more complicated yet universal and no change required by the users)

• Rearchitect the indexables DB data, such that the scheme/subdomain/domain/tld/port in the URL data are removed and overall irrelevant regardless of the environment, which benefits high portability in regards to environments/data-migration.
• Additionally, create a reusable class method containing a performant recomposition of the full URL based on the hostname stored in WordPress/server (or use one that already exists in WordPress already).
• Add this URL recomposition method to all places in the Yoast SEO plugin code where the full URL was normally returned.

For Both Above Solutions

• Document this change in the appropriate knowledgebase articles/blog posts.
• Educate your support staff in their responses that refer to the knowledgebase and subsequent correcting PR tickets.
• Add a notice in the Yoast headers for the next 2 releases to make users aware that might be affected of this change.
• Follow up with users in the original support tickets that had issues before, with a generic followup response that provides context for this change to revert the original PR and references the knowledgebase/blog posts.

Why do you think this feature is something we should consider for the Yoast SEO plugins?

The implemented solution in the PR has only traded out one set of issues and support requests in place of a new set of issues and support requests. Now users that stage their content on a staging environment (where writers, authors and editors commonly stage their material) have issues with their SEO meta data not being indexed, even though their website content migrations had properly handled URL/domain-name differences between staging and production environments.

The original support requests for the issue of URL/domain-name differences between environments stem from unhandled ETL steps required in a user's migration configuration and ultimately are not the responsibility of the Yoast SEO plugin to resolve.

Dev-ops migration strategies are out of scope of what the plugin's code should be concerned with, and the PR solution is opinionated in its orchestration and based on an unqualified requirement that a majority of users would benefit from this change (people doing migrations correctly weren't complaining, and thus no support tickets quantifying the need for a change).

Additional context

The performance benefits we gain from the indexables table is fully realized and powerful, but the option to turn it off/on and rebuild it should be available in the settings of the plugin, not hidden in background logic based on an environment variable.

If users are migrating content from staging to production environments, they should also fully educate themselves on the intricacies of data/content migration, what tables the data/content resides in, and what pitfalls may occur with domain name URLs. They should then solve those issues in their migration strategy, not in the SEO strategy by not expecting the Yoast SEO plugin to do that for them, but instead their migration plugins/tools/processes.