-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
Jenkins 2.462.3, Java 11.0.26, Tomcat 9.0.102
Hey Team,
We are seeing a memory leak issue on Jenkins after upgrading a set of plugins as part of security maintenance (late November)
| Component | Version |
|---|---|
| Jenkins | 2.462.3 (Was 2.426.3) |
| Tomcat | 9.0.102 (Was 9.0.85, we upgraded to the latest stable 9 version) |
| Java | 11.0.26 (Was 11.0.23) |
| Maven | 3.8.8 (Was 3.8.4) |
Attaching the current plugin-list (Please ignore the bfs)
Regular restarts brings it down, but with the service running for over 3 months, we are seeing a considerable increase in used (JVM Memory) and swap memory.
Our max heap size is set to 300GB (Our underlying VM gives upto 1280GB Memory). This is a production service and we have good number of jobs and hits to it. On an avg
Threads on [email protected]: Number = 645, Maximum = 1,426, Total started = 71,282,726
We suspect something holds up the memory and hence the GC is not able to release junk memory. There are frequent young GC runs that frees upto (10-40GB) with every run, the same revived when a concurrent/Full GC runs (around 5/6 runs in a span of ~5months).
Plugins upgraded after which we noticed the leak (Syntax - Plugin(Version Before) Version After)
1 bouncycastle-api (2.29) 2.30.1.77-225.v26ea_c9455fd9
2 config-file-provider (3.8.2) 953.v0432a_802e4d2
3 jackson2-api (2.15.3-366.vfe8d1fa_f8c87) 2.16.1-373.ve709c6871598
4 ldap (1.13) 725.v3cb_b_711b_1a_ef
5 pipeline-graph-analysis (1.2) 202.va_d268e64deb_3
6 pipeline-input-step (2.12) 468.va_5db_051498a_4
7 pipeline-model-api (1.9.2) 2.2118.v31fd5b_9944b_5
8 pipeline-model-definition (1.9.2) 2.2118.v31fd5b_9944b_5
9 pipeline-model-extensions (1.9.2) 2.2118.v31fd5b_9944b_5
10 pipeline-stage-tags-metadata (1.9.2) 2.2118.v31fd5b_9944b_5
11 pipeline-rest-api (2.9) 2.34
12 pipeline-stage-step (2.5) 305.ve96d0205c1c6
13 role-strategy (3.1.1) 717.v6a_69a_fe98974
14 workflow-basic-steps (2.23) 969.vc4ec3e4854b_f
15 okhttp-api (3.14.9) 4.9.3-108.v0feda04578cf
16 saml (1.1.7) 4.464.vea_cb_75d7f5e0
17 metrics (4.0.2.8.1) 4.2.18-442.v02e107157925
18 jaxb (2.3.8-1) 2.3.9-1
19 javax-mail-api (1.6.2-2) 1.6.2-6
20 jakarta-activation-api (2.0.1-1) 2.0.1-3
21 jakarta-mail-api (2.0.1-1) 2.0.1-3
22 json-path-api (New-Plugin) 2.8.0-5.v07cb_a_1ca_738c
23 prometheus (New-Plugin) 787.v52e8f47488fc
24 joda-time-api (New-Plugin) 2.12.5-5.v5495a_235fedf
I will share the heap-dump analysis to add more to the issue seen.
Regards
Hema
