LB: per-rule keepalive and timeouts, plus three HAProxy config fixes - #14131
LB: per-rule keepalive and timeouts, plus three HAProxy config fixes#14131bhouse-nexthop wants to merge 5 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14131 +/- ##
============================================
+ Coverage 3.71% 19.90% +16.19%
- Complexity 0 20166 +20166
============================================
Files 487 6371 +5884
Lines 41992 576967 +534975
Branches 7942 70644 +62702
============================================
+ Hits 1558 114838 +113280
- Misses 40208 449573 +409365
- Partials 226 12556 +12330
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
With keepalive enabled, an LB rule that also uses HTTP stickiness or SSL offload emitted "no option forceclose". HAProxy has rejected that keyword since 2.0, and the system VM has shipped 2.x for several releases: [ALERT] config : parsing [haproxy.cfg:22]: option 'forceclose' is not supported any more since HAProxy 2.0, please just remove it, or use 'option httpclose' if absolutely needed. It is a fatal parse error, so haproxy keeps running the previous config and the rule silently never takes effect. Replaced with "option http-keep-alive", which says the same thing and is valid on every version the system VM has shipped. Signed-off-by: Brad House <bhouse@nexthop.ai>
a8f56df to
4eed46e
Compare
The global and defaults sections are static arrays. Wrapping them in
Arrays.asList() and calling set() writes through to the array, so a value
from one LB rule stays there for every config generated afterwards:
- keepalive on any rule leaves "no option httpclose" in the defaults
for every later rule, in any network, in any account
- an idle timeout from one rule becomes the default for the next
- idleTimeout 0 blanks the timeouts permanently
Copy the arrays instead. Same output, no shared state.
Signed-off-by: Brad House <bhouse@nexthop.ai>
An LB rule with AppCookie stickiness emitted "appsession", which haproxy has refused since 1.6: [ALERT] config : 'appsession' is not supported anymore since HAProxy 1.6. [ALERT] config : Fatal errors found in configuration. The whole file is rejected, so every rule on that router keeps serving its previous config, not just the one with the policy. The VR advertises AppCookie as supported, so this is reachable from the API. A stick table on the cookie is the documented replacement: stick-table type string len <length> size 10k expire <holdtime> stick store-response res.cook(<name>) stick match req.cook(<name>) stick store-request req.cook(<name>) # only with request-learn The prefix and mode options have no equivalent and are now logged and ignored. They were never applied - the directive carrying them was rejected - so no working behaviour changes. Signed-off-by: Brad House <bhouse@nexthop.ai>
4eed46e to
6a432c2
Compare
apache#12586 made the idle timeout global. Keepalive is set on the network offering, at create time only, with no UI. Both land in the haproxy defaults section, so every rule on a router shares them. Three optional parameters on createLoadBalancerRule and updateLoadBalancerRule, each written to that rule's own listen section: | parameter | haproxy directive | unset | | keepalive | option http-keep-alive / httpclose | offering | | idletimeout | timeout client, timeout server | global | | keepalivetimeout | timeout http-keep-alive | idletimeout| Held as firewall rule details, so no schema change. A rule on port 80 can now have keepalive and X-Forwarded-For together, which no combination of the existing settings can produce: offering flag off mode http closes per response XFF works offering flag on mode tcp reuses connections XFF lost per rule mode http reuses connections XFF works A rule that sets nothing behaves exactly as before, including the fall back to tcp mode when the offering flag is on. Negative timeouts are rejected at the API and dropped in the generator. Haproxy treats one as a fatal parse error, which would strand every rule on the router. Applies to public LB rules. Application load balancers are created through a different command and are not covered. Signed-off-by: Brad House <bhouse@nexthop.ai>
6a432c2 to
08327da
Compare
|
@bhouse-nexthop |
haproxy_check.py compares the router's haproxy.cfg against what the management server thinks it configured. It knew about maxconn and the global idle timeout, so a rule's own keepalive and timeouts could drift without anything noticing. The health check data now carries the three per rule values, empty when the rule inherits, and the check validates them in that rule's listen section: ruleKeepAlive option http-keep-alive, or option httpclose ruleIdleTimeout timeout client, timeout server ruleKeepAliveTimeout timeout http-keep-alive The http mode test needed widening too. It read the offering flag alone, so a rule that sets keepalive itself stopped being checked at all: before port 80 and the offering has keepalive off after port 80 and (the rule sets keepalive, or the offering has it off) An older management server sends none of these keys, and a rule that sets nothing sends them empty. Both read as nothing to check. Signed-off-by: Brad House <bhouse@nexthop.ai>
|
@weizhouapache good catch, thanks - added in the latest commit.
The http mode test needed widening as well, otherwise a rule that sets keepalive itself stopped Compatibility: an older management server sends none of these keys, and a rule that inherits sends I ran the script against generated configs rather than only reading it - a matching config passes, |
Description
#12586 made the load balancer idle timeout globally configurable, as
network.loadbalancer.haproxy.idle.timeout. This expands that to per load balancer rule, anddoes the same for keepalive, which until now could only be set on the network offering.
Where these settings live today:
keepaliveenabled, create-time only, no UIoption httpcloseindefaultsnetwork.loadbalancer.haproxy.idle.timeout(#12574, #12586)timeout client,timeout serverindefaultsBoth land in the
defaultssection, so they apply to every LB rule on every network using thatoffering. One rule cannot differ from another on the same router.
New parameters on
createLoadBalancerRuleandupdateLoadBalancerRuleEach is optional, and each writes into that rule's own
listensection, which overridesdefaults:keepaliveoption http-keep-alive, elseoption httpclosekeepaliveenabledidletimeouttimeout clientandtimeout serverkeepalivetimeouttimeout http-keep-aliveidletimeoutappliesidletimeoutkeeps the global's meaning, including0for infinite.What that produces, taken from the generator - an http rule with all three set, a tcp rule with
only an idle timeout, and a rule that set nothing:
The three values are returned on
listLoadBalancerRules, and are settable on the load balancerrule's edit form in the UI, where blank reads as "Use default".
A negative timeout is rejected by the API. Haproxy treats one as a fatal parse error, which would
leave every rule on that router serving its previous config, so the generator drops it as well.
Stored as
firewall_rule_details. No schema change.Applies to public LB rules. Application load balancers are created through a different command and
are not covered.
What this makes possible
A rule on port 80 can now have keep-alive and
X-Forwarded-For. It could not before:mode httpConnection: closeper responsemode tcpforwardforneeds http modemode httpA rule that sets nothing keeps today's behaviour exactly, including the fall back to
mode tcpwhen the offering flag is on.
Also fixes three bugs in the same file
1.
option forcecloseis not a valid keywordWith keepalive enabled, a rule that also uses HTTP-based stickiness or SSL offload emits
no option forceclose. HAProxy has rejected that keyword since 2.0, and the system VM is Debian 12(HAProxy 2.6):
The whole file is rejected, so haproxy keeps serving the previous config and the rule silently
never takes effect. Replaced with
option http-keep-alive.2. One rule's settings leak into every rule generated afterwards
globalSectionanddefaultsSectionarestatic String[]. The generator wraps them inArrays.asList(...)and callsset(), which writes through to the arrays, so the valuesurvives the call:
no option httpclosein the defaults for every later rule, any network, any accountidleTimeout> 0idleTimeout== 0The generator now copies into a fresh list, and the sections themselves became
List.of(...), so afuture
set()throws instead of silently corrupting every config the agent builds afterwards.Output is unchanged for a single rule.
3.
appsessionwas removed from HAProxy in 1.6An LB rule with
AppCookiestickiness emittedappsession, which the shipped haproxy refuses:Same blast radius as bug 1 - the file is rejected, so every rule on that router keeps serving its
previous config. The VR advertises
AppCookieas a supported stickiness method, so this isreachable from the API.
Replaced with the documented stick table equivalent:
len <n>stick-table type string len <n> size 10ktimeout <h>expire <h>stick store-response res.cook(<name>)+stick match req.cook(<name>)request-learnstick store-request req.cook(<name>)prefix,modeprefixandmodewere never applied - the directive carrying them was rejected - so no workingbehaviour changes.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
How Has This Been Tested?
HAProxyConfiguratorTestTests run: 17, Failures: 0andLoadBalancingRulesManagerImplTestTests run: 16, Failures: 0.Each bug fix has a case that fails without it:
forceclose, and does emitoption http-keep-aliveno option httpclosebehindAppCookiestickiness emits a stick table and neverappsessionThe rest cover the per-rule settings:
timeout http-keep-aliveis emitted, and only when keepalive is onidletimeoutoverrides per rule on a tcp rule, leaving the defaults section aloneGenerated configs are checked against a real
haproxy -c, which is where the alert text abovecomes from. A file holding an
AppCookierule, an http rule with all three settings, a tcp rulewith a timeout, and a rule with a negative timeout is fatal before these changes and valid
after - the three alerts above become zero.