Memory leaks with nginx reload (without restart)

[martinhsv]

When, instead of restarting nginx, one performs a reload of the configuration, memory may leak.

The memory leaks are not large per reload, but if doing so frequently, the free memory reduction will become noticeable. A near-term mitigation is to at least periodically do a true restart.

This issue is being created in lieu of #2502 and others.

[Volatus]

@martinhsv Would it be possible to do a small patch release soon that would include the commit e9a7ba4 ? According to kubernetes/ingress-nginx#8166 (comment) it did alleviate the leaks that were significant in size and would be very helpful for the users for the time being as other leaks related to reloading get ironed out. Thanks.

[IanRobertson-wpe]

Hi @martinhsv, I believe my organization is encountering this issue as well. What is needed in order to address this? Is the root cause already identified?

[martinhsv]

Hello @Volatus ,

The ModSecurity project has not had a practice of doing patch releases for single issues that are separate from accumulated changes in v3/master. When the next 3.0.x release appears, it will almost certainly be from v3/master as that branch appears at that time.

The next release of 3.0.x is tentatively planned for the moderately soon period (by which I mean within 8-10 weeks) -- if that is within your definition of 'soon'. If that is problematically long for you, you could always build ModSecurity yourself from a point in time that includes the commit that you have referenced.

I do, understand, of course, that some installations may have a policy of only allowing use of official releases. If that is the case for you, and the aforementioned timeframe is problematic, I could take that into consideration. But that would have to be weighed against some other factors.

Hi @IanRobertson-wpe ,

Perhaps my response to Volatus answers most of what you are asking about as well.

One other thing I'll mention is that there is one (smaller) unresolved leak present with common ModSecurity usage. I have a fix for this for which I'll likely create the PR within a few days.

[Volatus]

Hello @Volatus ,

The ModSecurity project has not had a practice of doing patch releases for single issues that are separate from accumulated changes in v3/master. When the next 3.0.x release appears, it will almost certainly be from v3/master as that branch appears at that time.

The next release of 3.0.x is tentatively planned for the moderately soon period (by which I mean within 8-10 weeks) -- if that is within your definition of 'soon'. If that is problematically long for you, you could always build ModSecurity yourself from a point in time that includes the commit that you have referenced.

I do, understand, of course, that some installations may have a policy of only allowing use of official releases. If that is the case for you, and the aforementioned timeframe is problematic, I could take that into consideration. But that would have to be weighed against some other factors.

Hi @IanRobertson-wpe ,

Perhaps my response to Volatus answers most of what you are asking about as well.

One other thing I'll mention is that there is one (smaller) unresolved leak present with common ModSecurity usage. I have a fix for this for which I'll likely create the PR within a few days.

Thanks for the response. I think we may have a temporary fix that allows us to point to the specific commit that includes the memory leak fix. 8-10 weeks is not bad at all assuming our fix works for the time being.

[martinhsv]

There is a pull request available ( #2876 ) that resolves the most prominent remaining memory leak issue when doing an nginx reload (rather than restart).

The leak in this case is proportional to the number of '.conf' ModSecurity files being read in by the bison parser.

Users for whom this is a significant nuisance in the immediate period are invited to try out the PR during the period before it is merged.

[martinhsv]

Known memory leaks of general application that occur when executing nginx reload have been resolved as of v3.0.9.

I will, however, leave this item open for at least a few weeks to allow for any additional reports.

It is probable that any further memory leaks associated with reload are related to less-commonly used features. Specific information about what sorts of Sec* constructs appear to trigger the effect would be helpful.

[S0obi]

Hello @martinhsv, here is a report from our side.

Configuration : nginx 1.23.4 with modsecurity 3.0.9 (Ubuntu 20.04.6), coreruleset-3.3.4 with few custom rules. 197 servers blocks in nginx configs with 191 "modsecurity on" directives.

Here are few tests I performed and data I collected :

• Reloading nginx multiple times (systemctl reload nginx) with ModSecurity 3.0.9 will not consume more memory. With modsecurity < 3.0.9, multiple reload will increase the used amount of memory
• I restarted nginx (systemctl restart nginx), after few second the used memory usage will stabilize around 2.53Go. After a first reload, the memory will peak and stabilize at 4.53 Go. Second and third reload after will not change the used memory (I can only a short burst during reload to 4.62Go and then get back to 4.53Go)

However, I am still observing a rampant memory leak over time. Here is a screenshot of the memory usage of our production server (similar to test server):

This situation is not new to modsecurity 3.0.9 but was also noticeable in the previous versions.

[Zoey2936]

Known memory leaks of general application that occur when executing nginx reload have been resolved as of v3.0.9.

I still get one. Running nginx -s reload makes my server using all 2GB of Ram and 4GB of swap, also cpu goes up to 100% until the server freezes, killing and relaunching instead makes the server use 1,5GB ram and 3GB of swap with the cpu being at nearly 0%. I use CRS and the default modsec configuration.

[GNU-Plus-Windows-User]

I can also confirm the memory leak on reload still exists. personally, I've found the memory leak size is based on the number of modsecurity on; and modsecurity_rules_file /path/to/example.conf; directives and the use of @pmFromFile.