Elastic & server going OOM

[GioDeGruyter]

Is your question related to a problem? Please describe.
The server of our client is currently running out of memory due to high memory use of ElasticSearch. During the start of the sales we were having issues that Elastic was crashing with 7.8gb of memory. Elastic memory has since been increased to around 12gb and is still going out of memory at times. We're only having this issue now during the sales periode. This has never happened outside of a sales period. We're wondering if this is anything related to our catalog/category setup (we suspect the virtual category setup)? Our client uses quite some virtual categories. They mainly use these to have a subselection in the filtering of the main categories.

Additional context

• module version 2.10.22.2
• The catalog consists of around 40.000 products, which doesn't seem excessive.
• An example of the client's category tree is the following, with each category that has children containing one level of virtual categories to be able to distinguish between product types (as below "Essentials")

• The sales category is the exception here, which has a lot of nested virtual categories. All categories in the screenshot below, except for the sales category itself are virtual categories:

If the virtual category setup would be the issue, are there any plans on improving this in future releases? Or do you have any suggestions on how the issue could be resolved?

Kind regards,
Giovanni

[GioDeGruyter]

Kind reminder.

[romainruaud]

Hi,

You have to keep in mind that when computing category/product association for a high-level category (or the search result page), the engine will build a complex query to assemble all the products belonging to the currently viewed category:

• products belonging to non-virtual children of the current category
• products matching one of any virtual children of the current category

The resulting query could end up being quite complex and adding pressure to the computation mechanism done by Elasticsearch.

That’s why we strongly recommend not using them for the whole category tree.

That's not the purpose behind virtual categories: they are designed to build ephemeral categories, such as dynamic corners: "special father day", "actually discounted products", "new products" or a category based on something that will often change: discount, stock, total number of sales, etc…

So yeah, most probably all the levels of imbrications you're having under "Sales" could hurt, especially if they are using conditions based on categories, like "Category is one of X". These conditions should be replaced by other way to target products, like "attribute set = X".

Could you check how much virtual categories you're having exactly ?

[GioDeGruyter]

Hi Romain,

Thx for the explanation!
They currently have 127 active virtual categories.

[romainruaud]

Are they built with "Category is one of" clauses ?

[GioDeGruyter]

There's a total of 17 active categories which have category ids set as a virtual rule (366 in total but most are disabled). I suppose the disabled ones shouldn't be an issue?

[roman204]

@romainruaud Maybe a fix for that yould be #3486 ?
I also got a project where the method getQueryCache fills up the redis-cache with huge data which leads to a OOM exception.

files: vendor/smile/elasticsuite/src/module-elasticsuite-virtual-category/Helper/Rule.php,
vendor/smile/elasticsuite/src/module-elasticsuite-virtual-category/Model/Rule.php

method: \Smile\ElasticsuiteVirtualCategory\Model\Rule::getCategorySearchQuery

./redis-key-size.sh | sort -t':' -nr -k5 |head -n20 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2834_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2832_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2827_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2826_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2809_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2807_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2804_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2803_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2797_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2796_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2795_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2794_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2777_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2773_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2765_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2764_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2763_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2762_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2759_0_1 | MEMORY USAGE:2097592 KEY: zc:k:596_GETCATEGORYSEARCHQUERY_1_2757_0_1 | MEMORY USAGE:2097592

[vahonc]

@GioDeGruyter,

Could you provide a screenshot from the Indices page (Elasticsuite -> Indices)? We want to know the size of your indexes.

BR,
Vadym

[GioDeGruyter]

Hi @vahonc ,

Kr,
Giovanni