Magento 2.2.0rc23: Customer Grid Indexer not working #10838

jhruehl · 2017-09-11T13:35:51Z

The Customer Grid Indexer doesn't work, when reindexing via CLI.

Preconditions

local - Docker Container:

PHP 7.0 and 7.1 (default)
MySQL 5.5.5-10.1.25-MariaDB
Magento 2.2.0rc23

Steps to reproduce

Manage to upgrade your store via composer from 2.1.7 to 2.2.0rc23 (good luck here!)
Run setup:upgrade and fix all problems, which occur during the process until it runs without problems
Run reindex of Cutomer Grid (indexer:reindex customer_grid) via CLI command

Expected result

At this stage only problems occuring, which are caused by plugins from extern modules using deprecated functions, classes, etc.)

Actual result

VoidWalker · 2017-09-18T15:15:04Z

Same issue on 2.1.8 and 2.1.7

magento-engcom-team · 2017-09-19T13:18:39Z

@jhruehl thank you for your report. Unfortunately, we can't guarantee that 3-rd party modules will work with newest Magento versions correctly if they depend on internal Magento code and not on code which marked with @api.
Our team works hard on keeping further Magento versions from incompatible changes, but that's not always possible. If you'd like to update your issue, please reopen it.

jhruehl · 2017-09-20T14:56:52Z

Nonsense. There is no third party module involved. Checked it by deactivating them all. And even the stack trace makes it quite obvious that its not a third party module. And the Customer Grid Indexer works just fine until updating to 2.2.0rc3.0.

So my guess is that you changed the Customer Grid Reindexer process in a way and forgot about Customer Attributes, which have the 'visible in grid' option set. Because that is the most probable reason why the Customer Grid Indexer calls the Attribute Handler.

orlangur · 2017-09-20T15:26:15Z

@jhruehl yeah, I was wondering how it is identified as a third-party module issue from the screenshot as well. Is the issue still reproducible for you on latest 2.2.0-preview branch?

jhruehl · 2017-09-21T06:24:53Z

@orlangur Yes, it is. But I'm going to try to reproduce it now in our vanilla m2.2.0rc3 store test server by adding a customer attribute with 'visible in grid' set to yes.

jhruehl · 2017-09-21T07:59:15Z

Sadly I couldn't reproce the Cutomer Grid Indexer problem with just adding the same attribute to the vanilla store we have already implemented. So I need to investigate further by debugging. But it could be that the massive difference between our productive customer data (where some incoherence snuggled in over the years) and the few (actually two customers), which are set in the vanilla installation.

VoidWalker · 2017-09-21T08:58:49Z

There is a patch for this problem MDVA-4538_EE_2.1.4_v2
Which you can request through partner portal

orlangur · 2017-09-21T13:32:54Z

@VoidWalker EE_2.1.4 does not seem to be suitable for 2.2.0rc23. I believe such an old patch (2.1.4 - 2.1.9) is already included in 2.2.0.

jhruehl · 2017-09-21T13:42:42Z

I was actually able to reproduce it in our vanilla m2.2.0rc3.0 store. In my first run, I didn't set 'is_used_in_grid'. Going to open a new Issue, because otherwise it probably gets ignored.

orlangur · 2017-09-21T13:44:08Z

@jhruehl could you check the latest 2.2.0-preview please? I'll reopen an issue for you if reproducible.

jhruehl · 2017-09-21T13:46:41Z

@orlangur Isn't the 2.2.0rc3.0 Version the latest Preview?

orlangur · 2017-09-21T13:48:26Z

It is the last RC probably, but the branch was updated after that: https://github.com/magento/magento2/tree/2.2.0-preview

jhruehl · 2017-09-21T13:56:27Z

Just looked at the problematic classes in the 2.2.0 and the 2.2.1-preview. They haven't been updated since then.
'Magento\Customer\Model\Indexer\Source' and '/lib/internal/Magento/Framework/Indexer/Handler/AttributeHandler'

So I don't really see the point to invest a lot of time into an update process to a preview branch, if the function addAttributeToSelect() is still missing in the Source class and still called in the AttributeHandler class.

jhruehl · 2017-09-21T14:12:14Z

So even the folders '/lib/internal/Magento/Framework/Indexer/' and 'Magento/Customer/Model/Indexer/' were not updated for a month, which leads to the conclusion that going into a probably not composer supported update will be just a waste of time leading to the same results.

orlangur · 2017-09-21T14:13:16Z

Sure, makes sense to me.

orlangur · 2017-09-21T14:14:13Z

@magento-engcom-team please provide some insight on #10838 (comment), looks like the issue is reproducible on vanilla installation as well.

jhruehl · 2017-09-21T14:18:01Z

@orlangur Thanks a lot for your help.

jhruehl · 2017-09-21T14:20:08Z

@magento-engcom-team The key to reproduce the problem is setting up a customer eav attribute, which has the 'is_used_in_grid' option set to 1.

jhruehl · 2017-09-21T14:23:59Z

Here is the latest command line stack trace from our vanilla magento 2.2.0rc3.0 installation:
magento2_vanilla@d2372a84f1f8:/var/www/users/magento2_vanilla/src/scripts$ ./magento.sh indexer:reindex customer_grid

Fatal error: Uncaught Error: Call to undefined method Magento\Customer\Model\Indexer\Source::addAttributeToSelect() in /mnt/host/projects/clients/magento2-vanilla/src/www/lib/internal/Magento/Framework/Indexer/Handler/AttributeHandler.php:38
Stack trace:
#0 /mnt/host/projects/clients/magento2-vanilla/src/www/lib/internal/Magento/Framework/Indexer/Action/Base.php(310): Magento\Framework\Indexer\Handler\AttributeHandler->prepareSql(Object(Magento\Customer\Model\Indexer\Source), 'e', Array)
#1 /mnt/host/projects/clients/magento2-vanilla/src/www/lib/internal/Magento/Framework/Indexer/Action/Entity.php(26): Magento\Framework\Indexer\Action\Base->createResultCollection()
#2 /mnt/host/projects/clients/magento2-vanilla/src/www/lib/internal/Magento/Framework/Indexer/Action/Base.php(179): Magento\Framework\Indexer\Action\Entity->prepareDataSource(Array)
#3 /mnt/host/projects/clients/magento2-vanilla/src/www/lib/internal/Magento/Framework/Indexer/Action/Base.php(189): Magento\Framework\Indexer\Action\Base->execute()
#4 /mnt/host/pro in /mnt/host/projects/clients/magento2-vanilla/src/www/lib/internal/Magento/Framework/Indexer/Handler/AttributeHandler.php on line 38

okorshenko · 2017-09-21T15:10:58Z

@jhruehl @orlangur thank you for research. we will take a look at this

okorshenko · 2017-09-21T15:11:34Z

@jhruehl it would be great if you can submit pull request if you know how to fix the issue

jhruehl · 2017-09-22T07:32:39Z

@okobchenko The solution seems not be just a simple fix, but looking at the code structure a public function addAttributeToSelect() needs to be implemented in the Magento\Customer\Model\Indexer\Source class as there a similar functions like addFieldToSelect() implemented by passing the requests to other injected classes, which have the needed functionality already implemented.

Though it seems that the by dependency injection reachable addAttributeToSelect() functions don't work without further adjustments in that case, because I tried to simply copy them into the problemtatic Source class without real adjustments and this mostly lead to the attribute not being found, because the right tables to join with weren't addressed.

But it seems a good approach would be similar to Magento/Catalog/Model/ResourceModel/Category/Flat/Collection::addAttributeToSelect(), but as I already stated needs to be adjusted to join with the right tables and probably some more.

VoidWalker · 2017-09-22T07:41:04Z

@okorshenko I feel like it make sense to repeat. There is a patch for this issue MDVA-4538_EE_2.1.4_v2
It might be not totally suitable fir 2.2.* but it shows the fix idea

jhruehl · 2017-09-28T11:02:39Z

Here ist the patch that solves the problem. Though I don't really get why Magento did leave this function out. And if you look at the content of the MDVA-4538_EE_2.1.4_v2 patch, it seems that the 2.1.4 EE version of Magento already used the same indexer logic as CE 2.2 uses now.

It would be really interesting to know what the thought behind that different implementation was.

fix_customer_grid_indexer_add_missing_function 2.txt

cotardo18 · 2018-01-23T20:40:38Z

@agata-maksymiuk I agree with you, having an extra attribute on customer's grid is possible only if the attribute type is changed in eav table to static

cotardo18 · 2018-01-23T23:38:00Z

@pfortin-expertime answer from @selusi is the good one! if you take my answer you won't be able to save data

hostep · 2018-01-24T09:04:46Z

Just FYI: I've got confirmation from the Magento Cloud support that this issue should get fixed in Magento 2.2.4

The fix should be:

diff --git a/vendor/magento/framework/Indexer/Handler/AttributeHandler.php b/vendor/magento/framework/Indexer/Handler/AttributeHandler.php
index 94f34ec..4b74ce5 100644
--- a/vendor/magento/framework/Indexer/Handler/AttributeHandler.php
+++ b/vendor/magento/framework/Indexer/Handler/AttributeHandler.php
@@ -35,7 +35,7 @@ class AttributeHandler implements HandlerInterface
                 'left'
             );
         } else {
-            $source->addAttributeToSelect($fieldInfo['origin'], 'left');
+            $source->addFieldToSelect($fieldInfo['origin'], 'left');
         }
     }
 }

Looks like there is a pending PR which should fix this: #12729

Fixed issues: - MAGETWO-81802: Magento 2.2.0rc23: Customer Grid Indexer not working #10838 - MAGETWO-84448: Product Import does not allow store-specific Custom Option labels - MAGETWO-85207: Customer's order grid is not displaying - MAGETWO-86657: Import existing customer with only three columns will override customer group_id and store_id - MAGETWO-86154: Wrong page cached for logged in user - MAGETWO-83293: Inefficient SQL query on applying filter on sales order grid - MAGETWO-83343: Customer shopping cart not accessible in admin order - MAGETWO-85708: Zoom the image can't be closed by iPhone with Safari browser

Fixed issues: - MAGETWO-81802: Magento 2.2.0rc23: Customer Grid Indexer not working magento#10838 - MAGETWO-84448: Product Import does not allow store-specific Custom Option labels - MAGETWO-85207: Customer's order grid is not displaying - MAGETWO-86657: Import existing customer with only three columns will override customer group_id and store_id - MAGETWO-86154: Wrong page cached for logged in user - MAGETWO-83293: Inefficient SQL query on applying filter on sales order grid - MAGETWO-83343: Customer shopping cart not accessible in admin order - MAGETWO-85708: Zoom the image can't be closed by iPhone with Safari browser MAGETWO-87132: Remove JsTestDriver from Magento codebase MAGETWO-84219: Creditmemo comment via REST api does not send creditmemo update email MAGETWO-84219: Creditmemo comment via REST api does not send creditmemo update email MAGETWO-86452: [Backport for 2.2.x] Update copyright year alignement array assignement For Your Eyes Only Remove whitespace at end of line FIX: out-of-stock options for configurable product visible on frontend as sellable Adapt \Magento\Analytics\Block\Adminhtml\System\Config\CollectionTimeLabel::render method to use current locale to render time zone label, as done in \Magento\Framework\Locale\TranslatedLists::getOptionTimezones Handle transparncy correctly for watermark The watermark functionality uses imagecopymerge for copying the watermark into the image, this function loses the alpha information. Therefor use imagecopy for 100% opacity and use imagefilter in other cases. Resolves: magento#10661 Fix handling of watermark with alpha for Imagick When adding a watermark that already has an alpha channel this was reset because the overal opacity was set for the image. Instead we should multiply the original alpha with the requested opacity Handle non true color images with GD2 Add integration test for watermark with alpha Added integration tests for handling alpha transparency in watermarks. Renamed the current test for watermarks as it only tests if the watermark is correctly places Fix issues with static tests Remove ui-state-active from siblings of the expanded menu item Previously when a menu item was expanded the class `ui-state-active` was not removed from the previous expanded menu item. This resulted in two (or more if you expanded more) menu items with this class. Clean region when select country :bug: add fallback for Product_links position attribute if position index was not set in API request :ok_hand: shortening variable name :bug: fix variable re-naming :ok_hand: drop multiple empty lines :ok_hand: closing parenthesis and the opening brace of a multi-line function on the same line :ok_hand: closing parenthesis and the opening brace of a multi-line function on the same line :ok_hand: reset protected variable naming :ok_hand: rename isPositionsSet function to isPositionSet to keep consistency Fix typo :ok_hand: apply code enhancement :ok_hand: apply code enhancement Add type hint

okorshenko · 2018-02-27T22:19:05Z

The issue has been fixed in 7759827 and delivered in 25e3157

jhruehl · 2018-02-28T08:29:26Z

Just updated to Magento 2.2.3 CE and it's not fixed. Had to apply my patch to make the customer grid indexer work again.
`scripts/magento.sh indexer:reindex customer_grid

Fatal error: Uncaught Error: Call to undefined method Magento\Customer\Model\Indexer\Source::addAttributeToSelect() in /mnt/host/projects/clients/qualipet/src/www/vendor/magento/framework/Indexer/Handler/AttributeHandler.php:38
Stack trace:
#0 /mnt/host/projects/clients/qualipet/src/www/vendor/magento/framework/Indexer/Action/Base.php(310): Magento\Framework\Indexer\Handler\AttributeHandler->prepareSql(Object(Magento\Customer\Model\Indexer\Source), 'e', Array)
#1 /mnt/host/projects/clients/qualipet/src/www/vendor/magento/framework/Indexer/Action/Entity.php(26): Magento\Framework\Indexer\Action\Base->createResultCollection()
#2 /mnt/host/projects/clients/qualipet/src/www/vendor/magento/framework/Indexer/Action/Base.php(179): Magento\Framework\Indexer\Action\Entity->prepareDataSource(Array)
#3 /mnt/host/projects/clients/qualipet/src/www/vendor/magento/framework/Indexer/Action/Base.php(189): Magento\Framework\Indexer\Action\Base->execute()
#4 /mnt/host/projects/clients/qualipet/src/www/vendor/magento/module-indexer/Model/Ind in /mnt/host/projects/clients/qualipet/src/www/vendor/magento/framework/Indexer/Handler/AttributeHandler.php on line 38`

hostep · 2018-02-28T08:34:10Z

@jhruehl: I'm pretty sure the fix is scheduled for 2.2.4 (see my latest comment), 2.2.3 only fixes security issues, no other bug fixes as far as I can see.

jhruehl · 2018-02-28T08:34:11Z

Here's the patch again for everyone, who still needs a fix, because they have a custom customer attribute, which is used in grid, in their store.

fix_customer_grid_indexer_add_missing_function.txt

jhruehl · 2018-02-28T08:41:54Z

@hostep Doesn't really matter, if the issue is flagged with 'fixed in 2.2.x'. If they flagged the issue to be fixed, but the fix isn't even given in an update afterwards, the flag is quite misleading, isn't it? Then they should use a 'fixed in 2.2.4' flag and given that, I wouldn't even have reported the error. Again.

roobware · 2018-03-01T07:39:47Z

A Patch in the Core Files is not best idea. You can made a module with this patch.

Yesterday, i made a moudle quickly, to fix this problem. You can find it on composer require vitd/m2_fixes.

Do not forget do following:

bin/magento setup:upgrade
bin/magento setup:di:compile
bin/magento cache:clean
bin/magento cache:flush
bin/magento setup:static-content:deploy <languages with spaces> -f (on cli will not content deployed in all store used languages, you can use alternative change the deploy:mode:set (developer, production) and back, this work proper).

You can modifying the module on your own wish or you can write an issue ticket on the repo page.

I hope this will help any one :)

magento/magento2#10838

victortodoran · 2018-04-16T10:48:05Z

Still broken in 2.2.3 Commerce.

himanshu8dev · 2018-04-16T11:51:15Z

Yes!, I have downloaded and upgrade magento 2.2.2 to 2.2.3. but still issue is not fixed from core.

t-richards · 2018-05-08T19:33:55Z

I can confirm this issue has been fixed in Magento Open Source 2.2.4.

khasru · 2018-05-17T11:48:48Z

In my case when i create a custom customer attribute and "Add to Column Options" to yes in magento2 EE. i got this error when i try reindex.
So that i set this field value to no. and its working fine.

Magento Team needs to taking care it.

hafizjee · 2018-09-18T07:00:02Z

i have add this function public function addAttributeToSelect($attribute, $joinType = false) { $this->customerCollection->addAttributeToSelect($attribute, $joinType); return $this; } in app/code/Magento/Customer/Model/Indexer/Source.php
but got an error while re indexing.
PHP Fatal error: Uncaught Error: Call to undefined method Magento\Customer\Model\Indexer\Source::addAttributeToSelect() in /var/www/html/ncsmokehouse/vendor/magento/framework/Indexer/Handler/AttributeHandler.php:38
what m i do ?

selusi · 2018-09-18T07:38:01Z

i have add this function public function addAttributeToSelect($attribute, $joinType = false) { $this->customerCollection->addAttributeToSelect($attribute, $joinType); return $this; } in app/code/Magento/Customer/Model/Indexer/Source.php
but got an error while re indexing.
PHP Fatal error: Uncaught Error: Call to undefined method Magento\Customer\Model\Indexer\Source::addAttributeToSelect() in /var/www/html/ncsmokehouse/vendor/magento/framework/Indexer/Handler/AttributeHandler.php:38
what m i do ?

Before updating the index you have to launch the following commands:

php bin/magento setup:upgrade
php bin/magento setup:di:compile

hafizjee · 2018-09-18T07:41:53Z

i have add this function public function addAttributeToSelect($attribute, $joinType = false) { $this->customerCollection->addAttributeToSelect($attribute, $joinType); return $this; } in app/code/Magento/Customer/Model/Indexer/Source.php
but got an error while re indexing.
PHP Fatal error: Uncaught Error: Call to undefined method Magento\Customer\Model\Indexer\Source::addAttributeToSelect() in /var/www/html/ncsmokehouse/vendor/magento/framework/Indexer/Handler/AttributeHandler.php:38
what m i do ?

Before updating the index you have to launch the following commands:
php bin/magento setup:upgrade
php bin/magento setup:di:compile

i already launch these commands. but no success

hafizjee · 2018-09-18T13:28:40Z

I apologize because in the previous comment I indicated that the
Magento \ Customer \ Model \ Indexer \ Source.php file did not exist, while it is actually alive and well.

However, by modifying the indexer.xml file the index rebuilding work fine.

Alternativly is possible add the below function to the file Source.php:
public function addAttributeToSelect($attribute, $joinType = false)
{
    $this->customerCollection->addAttributeToSelect($attribute, $joinType);
    return $this;
}
I tested both methods on my test site.

m follow this but got an error . how can i resolve ?

hostep · 2018-09-18T14:59:59Z

@hafizjee, the correct fix is this one: 7759827

If you use composer, you can follow this guide to install this as a patch on top of your installation: https://support.magento.com/hc/en-us/articles/360005484154-Create-a-patch-for-a-Magento-2-Composer-installation-from-a-GitHub-commit

Or you can just upgrade to Magento 2.2.4 or higher where this fix is included.

selusi · 2018-09-18T15:00:57Z

@hafizjee
You can try to delete the folders:
generated\code
var\cache
and then launch:

php bin/magento setup:upgrade
php bin/magento setup:di:compile

The error code leaves no doubt of interpretation. The application does not find the required method, so either you copied the code to the wrong file or you did not completely delete the cache.
Another possible cause could be the permissions assigned to the folders that could prevent the proper rewriting of the cache.

matinict · 2018-09-30T07:07:41Z

Here's the patch again for everyone, who still needs a fix, because they have a custom customer attribute, which is used in grid, in their store.

fix_customer_grid_indexer_add_missing_function.txt

work for me

gjportegies · 2018-11-14T08:52:34Z

I'm experiencing this problem on my Magento 2.1.12 as well. Is there any fix for the Magento 2.1.* ? Because the fixes mentioned earlier are all applying for Magento 2.2+

magento-engcom-team added Issue: Format is valid Gate 1 Passed. Automatic verification of issue format passed Issue: Clear Description Gate 2 Passed. Manual verification of the issue description passed labels Sep 11, 2017

magento-engcom-team closed this as completed Sep 19, 2017

magento-engcom-team added the non-issue label Sep 19, 2017

orlangur reopened this Sep 21, 2017

orlangur removed the non-issue label Sep 21, 2017

okorshenko closed this as completed Feb 27, 2018

okorshenko added the Fixed in 2.2.x The issue has been fixed in 2.2 release line label Feb 27, 2018

unicoder88 added a commit to ConvertGroupsAS/magento2-patches that referenced this issue Mar 20, 2018

Fix issue #10838 Customer Grid Indexer not working

c4ea03c

magento/magento2#10838

magento-engcom-team pushed a commit that referenced this issue Apr 11, 2018

MAGETWO-90109: Customer Grid Indexer not working #10838

4a2284c

magento-engcom-team pushed a commit that referenced this issue Apr 11, 2018

MAGETWO-90109: Customer Grid Indexer not working #10838

0933c7a

magento-engcom-team pushed a commit that referenced this issue Apr 11, 2018

MAGETWO-90109: Customer Grid Indexer not working #10838

9169132

JesKingDev added a commit to JesKingDev/magento2-patches that referenced this issue Apr 11, 2018

Patch for magento/magento2#10838 in 2.2.2 environment

f2f54f4

Magento 2.2.0rc23: Customer Grid Indexer not working #10838

Magento 2.2.0rc23: Customer Grid Indexer not working #10838

Comments

jhruehl commented Sep 11, 2017

Preconditions

Steps to reproduce

Expected result

Actual result

VoidWalker commented Sep 18, 2017

magento-engcom-team commented Sep 19, 2017

jhruehl commented Sep 20, 2017 • edited Loading

orlangur commented Sep 20, 2017

jhruehl commented Sep 21, 2017

jhruehl commented Sep 21, 2017

VoidWalker commented Sep 21, 2017

orlangur commented Sep 21, 2017

jhruehl commented Sep 21, 2017

orlangur commented Sep 21, 2017

jhruehl commented Sep 21, 2017

orlangur commented Sep 21, 2017

jhruehl commented Sep 21, 2017

jhruehl commented Sep 21, 2017 • edited Loading

orlangur commented Sep 21, 2017

orlangur commented Sep 21, 2017

jhruehl commented Sep 21, 2017

jhruehl commented Sep 21, 2017

jhruehl commented Sep 21, 2017 • edited Loading

okorshenko commented Sep 21, 2017

okorshenko commented Sep 21, 2017

jhruehl commented Sep 22, 2017 • edited Loading

VoidWalker commented Sep 22, 2017

jhruehl commented Sep 28, 2017 • edited Loading

cotardo18 commented Jan 23, 2018

cotardo18 commented Jan 23, 2018

hostep commented Jan 24, 2018

okorshenko commented Feb 27, 2018

jhruehl commented Feb 28, 2018

hostep commented Feb 28, 2018

jhruehl commented Feb 28, 2018

jhruehl commented Feb 28, 2018

roobware commented Mar 1, 2018 • edited Loading

victortodoran commented Apr 16, 2018 • edited Loading

himanshu8dev commented Apr 16, 2018

t-richards commented May 8, 2018

khasru commented May 17, 2018

hafizjee commented Sep 18, 2018

selusi commented Sep 18, 2018

hafizjee commented Sep 18, 2018

hafizjee commented Sep 18, 2018

hostep commented Sep 18, 2018

selusi commented Sep 18, 2018 • edited Loading

matinict commented Sep 30, 2018

gjportegies commented Nov 14, 2018

jhruehl commented Sep 20, 2017 •

edited

Loading

jhruehl commented Sep 21, 2017 •

edited

Loading

jhruehl commented Sep 21, 2017 •

edited

Loading

jhruehl commented Sep 22, 2017 •

edited

Loading

jhruehl commented Sep 28, 2017 •

edited

Loading

roobware commented Mar 1, 2018 •

edited

Loading

victortodoran commented Apr 16, 2018 •

edited

Loading

selusi commented Sep 18, 2018 •

edited

Loading