Skip to content

LB: per-rule keepalive and timeouts, plus three HAProxy config fixes - #14131

Open
bhouse-nexthop wants to merge 5 commits into
apache:mainfrom
bhouse-nexthop:haproxy-keepalive
Open

LB: per-rule keepalive and timeouts, plus three HAProxy config fixes#14131
bhouse-nexthop wants to merge 5 commits into
apache:mainfrom
bhouse-nexthop:haproxy-keepalive

Conversation

@bhouse-nexthop

@bhouse-nexthop bhouse-nexthop commented Sep 10, 2026

Copy link
Copy Markdown
Collaborator

Description

#12586 made the load balancer idle timeout globally configurable, as
network.loadbalancer.haproxy.idle.timeout. This expands that to per load balancer rule, and
does the same for keepalive, which until now could only be set on the network offering.

Where these settings live today:

setting scope set where reaches haproxy as
keepalive network offering keepaliveenabled, create-time only, no UI option httpclose in defaults
idle timeout global network.loadbalancer.haproxy.idle.timeout (#12574, #12586) timeout client, timeout server in defaults

Both land in the defaults section, so they apply to every LB rule on every network using that
offering. One rule cannot differ from another on the same router.

New parameters on createLoadBalancerRule and updateLoadBalancerRule

Each is optional, and each writes into that rule's own listen section, which overrides defaults:

parameter type haproxy directive applies to unset means
keepalive boolean option http-keep-alive, else option httpclose rules served in http mode the offering's keepaliveenabled
idletimeout long, ms timeout client and timeout server every rule, tcp and http the global from #12586
keepalivetimeout long, ms timeout http-keep-alive http mode, keepalive on idletimeout applies

idletimeout keeps the global's meaning, including 0 for infinite.

What that produces, taken from the generator - an http rule with all three set, a tcp rule with
only an idle timeout, and a rule that set nothing:

listen 10_2_0_2-80
	bind 10.2.0.2:80
	mode http
	option http-keep-alive
	timeout http-keep-alive 15000
	timeout client     600000
	timeout server     600000
	balance roundrobin
	server 10_2_0_2-80_0 10.1.10.3:80 check

listen 10_2_0_3-3306
	bind 10.2.0.3:3306
	timeout client     600000
	timeout server     600000
	balance roundrobin
	server 10_2_0_3-3306_0 10.1.10.4:3306 check

listen 10_2_0_4-8080
	bind 10.2.0.4:8080
	balance roundrobin
	server 10_2_0_4-8080_0 10.1.10.5:8080 check

The three values are returned on listLoadBalancerRules, and are settable on the load balancer
rule's edit form in the UI, where blank reads as "Use default".

A negative timeout is rejected by the API. Haproxy treats one as a fatal parse error, which would
leave every rule on that router serving its previous config, so the generator drops it as well.

Stored as firewall_rule_details. No schema change.

Applies to public LB rules. Application load balancers are created through a different command and
are not covered.

What this makes possible

A rule on port 80 can now have keep-alive and X-Forwarded-For. It could not before:

keepalive mode connection reuse XFF
offering flag off (the default) mode http no, Connection: close per response yes
offering flag on mode tcp yes no, forwardfor needs http mode
per rule, this PR mode http yes yes

A rule that sets nothing keeps today's behaviour exactly, including the fall back to mode tcp
when the offering flag is on.

Also fixes three bugs in the same file

1. option forceclose is not a valid keyword

With keepalive enabled, a rule that also uses HTTP-based stickiness or SSL offload emits
no option forceclose. HAProxy has rejected that keyword since 2.0, and the system VM is Debian 12
(HAProxy 2.6):

[ALERT] config : parsing [haproxy.cfg:22]: option 'forceclose' is not supported any more
        since HAProxy 2.0, please just remove it, or use 'option httpclose' if absolutely
        needed.
[ALERT] config : Fatal errors found in configuration.

The whole file is rejected, so haproxy keeps serving the previous config and the rule silently
never takes effect. Replaced with option http-keep-alive.

2. One rule's settings leak into every rule generated afterwards

globalSection and defaultsSection are static String[]. The generator wraps them in
Arrays.asList(...) and calls set(), which writes through to the arrays, so the value
survives the call:

set once by leaks as
keepalive on any rule no option httpclose in the defaults for every later rule, any network, any account
idleTimeout > 0 that timeout becomes the default for the next rule
idleTimeout == 0 timeouts blanked permanently

The generator now copies into a fresh list, and the sections themselves became List.of(...), so a
future set() throws instead of silently corrupting every config the agent builds afterwards.
Output is unchanged for a single rule.

3. appsession was removed from HAProxy in 1.6

An LB rule with AppCookie stickiness emitted appsession, which the shipped haproxy refuses:

[ALERT] config : 'appsession' is not supported anymore since HAProxy 1.6.
[ALERT] config : Fatal errors found in configuration.

Same blast radius as bug 1 - the file is rejected, so every rule on that router keeps serving its
previous config. The VR advertises AppCookie as a supported stickiness method, so this is
reachable from the API.

Replaced with the documented stick table equivalent:

appsession replacement
len <n> stick-table type string len <n> size 10k
timeout <h> expire <h>
the cookie stick store-response res.cook(<name>) + stick match req.cook(<name>)
request-learn stick store-request req.cook(<name>)
prefix, mode no equivalent, logged and ignored

prefix and mode were never applied - the directive carrying them was rejected - so no working
behaviour changes.

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

How Has This Been Tested?

HAProxyConfiguratorTest Tests run: 17, Failures: 0 and LoadBalancingRulesManagerImplTest
Tests run: 16, Failures: 0.

Each bug fix has a case that fails without it:

  • keepalive with SSL offload emits no forceclose, and does emit option http-keep-alive
  • a keepalive config followed by a non-keepalive one leaves no no option httpclose behind
  • an idle timeout followed by an unset one falls back to the built-in default
  • AppCookie stickiness emits a stick table and never appsession

The rest cover the per-rule settings:

  • a rule that sets keepalive stays in http mode
  • a rule's keepalive wins over the offering, in both directions
  • a rule that sets nothing is byte for byte what it was before
  • timeout http-keep-alive is emitted, and only when keepalive is on
  • idletimeout overrides per rule on a tcp rule, leaving the defaults section alone
  • a negative timeout never reaches the config, and is rejected at the API
  • two rules in one config do not pick up each other's settings

Generated configs are checked against a real haproxy -c, which is where the alert text above
comes from. A file holding an AppCookie rule, an http rule with all three settings, a tcp rule
with a timeout, and a rule with a negative timeout is fatal before these changes and valid
after
- the three alerts above become zero.

@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 35.06494% with 100 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.90%. Comparing base (4ce5291) to head (c8cf0fd).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...a/com/cloud/network/router/CommandSetupHelper.java 0.00% 18 Missing ⚠️
...in/java/com/cloud/network/HAProxyConfigurator.java 70.73% 9 Missing and 3 partials ⚠️
...src/main/java/com/cloud/api/ApiResponseHelper.java 0.00% 12 Missing ⚠️
...loud/network/lb/LoadBalancingRulesManagerImpl.java 52.00% 7 Missing and 5 partials ⚠️
...ork/router/VirtualNetworkApplianceManagerImpl.java 0.00% 12 Missing ⚠️
...d/user/loadbalancer/CreateLoadBalancerRuleCmd.java 0.00% 10 Missing ⚠️
...in/java/com/cloud/agent/api/to/LoadBalancerTO.java 50.00% 9 Missing ⚠️
.../cloudstack/api/response/LoadBalancerResponse.java 0.00% 9 Missing ⚠️
...d/user/loadbalancer/UpdateLoadBalancerRuleCmd.java 33.33% 6 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##              main   #14131       +/-   ##
============================================
+ Coverage     3.71%   19.90%   +16.19%     
- Complexity       0    20166    +20166     
============================================
  Files          487     6371     +5884     
  Lines        41992   576967   +534975     
  Branches      7942    70644    +62702     
============================================
+ Hits          1558   114838   +113280     
- Misses       40208   449573   +409365     
- Partials       226    12556    +12330     
Flag Coverage Δ
uitests 3.70% <ø> (-0.01%) ⬇️
unittests 21.17% <35.06%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bhouse-nexthop
bhouse-nexthop marked this pull request as draft September 10, 2026 15:03
With keepalive enabled, an LB rule that also uses HTTP stickiness or SSL
offload emitted "no option forceclose". HAProxy has rejected that keyword
since 2.0, and the system VM has shipped 2.x for several releases:

  [ALERT] config : parsing [haproxy.cfg:22]: option 'forceclose' is not
  supported any more since HAProxy 2.0, please just remove it, or use
  'option httpclose' if absolutely needed.

It is a fatal parse error, so haproxy keeps running the previous config
and the rule silently never takes effect.

Replaced with "option http-keep-alive", which says the same thing and is
valid on every version the system VM has shipped.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@bhouse-nexthop
bhouse-nexthop changed the base branch from 4.22 to main September 10, 2026 15:10
@bhouse-nexthop bhouse-nexthop changed the title Fix two bugs in the virtual router's HAProxy config LB: per-rule keepalive and timeouts, plus two HAProxy config fixes Sep 10, 2026
The global and defaults sections are static arrays. Wrapping them in
Arrays.asList() and calling set() writes through to the array, so a value
from one LB rule stays there for every config generated afterwards:

  - keepalive on any rule leaves "no option httpclose" in the defaults
    for every later rule, in any network, in any account
  - an idle timeout from one rule becomes the default for the next
  - idleTimeout 0 blanks the timeouts permanently

Copy the arrays instead. Same output, no shared state.

Signed-off-by: Brad House <bhouse@nexthop.ai>
An LB rule with AppCookie stickiness emitted "appsession", which haproxy
has refused since 1.6:

  [ALERT] config : 'appsession' is not supported anymore since HAProxy 1.6.
  [ALERT] config : Fatal errors found in configuration.

The whole file is rejected, so every rule on that router keeps serving
its previous config, not just the one with the policy. The VR advertises
AppCookie as supported, so this is reachable from the API.

A stick table on the cookie is the documented replacement:

  stick-table type string len <length> size 10k expire <holdtime>
  stick store-response res.cook(<name>)
  stick match req.cook(<name>)
  stick store-request req.cook(<name>)      # only with request-learn

The prefix and mode options have no equivalent and are now logged and
ignored. They were never applied - the directive carrying them was
rejected - so no working behaviour changes.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@bhouse-nexthop
bhouse-nexthop marked this pull request as ready for review September 10, 2026 16:04
@bhouse-nexthop bhouse-nexthop changed the title LB: per-rule keepalive and timeouts, plus two HAProxy config fixes LB: per-rule keepalive and timeouts, plus three HAProxy config fixes Sep 10, 2026
apache#12586 made the idle timeout global. Keepalive is set on the network
offering, at create time only, with no UI. Both land in the haproxy
defaults section, so every rule on a router shares them.

Three optional parameters on createLoadBalancerRule and
updateLoadBalancerRule, each written to that rule's own listen section:

  | parameter        | haproxy directive                    | unset      |
  | keepalive        | option http-keep-alive / httpclose   | offering   |
  | idletimeout      | timeout client, timeout server       | global     |
  | keepalivetimeout | timeout http-keep-alive              | idletimeout|

Held as firewall rule details, so no schema change.

A rule on port 80 can now have keepalive and X-Forwarded-For together,
which no combination of the existing settings can produce:

  offering flag off   mode http   closes per response   XFF works
  offering flag on    mode tcp    reuses connections    XFF lost
  per rule            mode http   reuses connections    XFF works

A rule that sets nothing behaves exactly as before, including the fall
back to tcp mode when the offering flag is on.

Negative timeouts are rejected at the API and dropped in the generator.
Haproxy treats one as a fatal parse error, which would strand every rule
on the router.

Applies to public LB rules. Application load balancers are created
through a different command and are not covered.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@weizhouapache weizhouapache added this to the 24.0 milestone Sep 10, 2026
@weizhouapache

Copy link
Copy Markdown
Member

@bhouse-nexthop
could you make some changes for VR health check ?
similar to #12596 which fixes #12591

haproxy_check.py compares the router's haproxy.cfg against what the
management server thinks it configured. It knew about maxconn and the
global idle timeout, so a rule's own keepalive and timeouts could drift
without anything noticing.

The health check data now carries the three per rule values, empty when
the rule inherits, and the check validates them in that rule's listen
section:

  ruleKeepAlive         option http-keep-alive, or option httpclose
  ruleIdleTimeout       timeout client, timeout server
  ruleKeepAliveTimeout  timeout http-keep-alive

The http mode test needed widening too. It read the offering flag alone,
so a rule that sets keepalive itself stopped being checked at all:

  before   port 80 and the offering has keepalive off
  after    port 80 and (the rule sets keepalive, or the offering has it off)

An older management server sends none of these keys, and a rule that sets
nothing sends them empty. Both read as nothing to check.

Signed-off-by: Brad House <bhouse@nexthop.ai>
@boring-cyborg boring-cyborg Bot added component:virtual-router Python Warning... Python code Ahead! labels Sep 10, 2026
@bhouse-nexthop

Copy link
Copy Markdown
Collaborator Author

@weizhouapache good catch, thanks - added in the latest commit.

haproxy_check.py knew about maxconn and the global idle timeout, so the new per rule settings
could drift with nothing noticing. The health check data now carries all three, and the check
validates them in that rule's own listen section:

data checked
ruleKeepAlive option http-keep-alive, or option httpclose
ruleIdleTimeout timeout client, timeout server
ruleKeepAliveTimeout timeout http-keep-alive

The http mode test needed widening as well, otherwise a rule that sets keepalive itself stopped
being checked at all:

before   port 80 and the offering has keepalive off
after    port 80 and (the rule sets keepalive, or the offering has it off)

Compatibility: an older management server sends none of these keys, and a rule that inherits sends
them empty. Both read as nothing to check, so an updated system VM template against an older
management server still passes.

I ran the script against generated configs rather than only reading it - a matching config passes,
and each of a wrong keepalive option, a drifted timeout client and a missing
timeout http-keep-alive fails.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants