Skip to content

[Bug] HttpServer double-closes accepted socket fd per request, causing unintended close of another thread's fd #747

Description

@aoshi2000

Summary

HttpServer::onConnect() closes the accepted socket fd multiple times per HTTP request.
In a multithreaded application, this can destroy file descriptors belonging to other threads.

Expected Behavior

HttpConnection::disconnect() should close the socket fd exactly once per request.
Other threads' file descriptors should not be affected.

Actual Behavior

disconnect() closes the same fd 2-3 times per request.
In a multithreaded application, this causes a race condition where another thread's fd (obtained via socket()/pipe() reusing the freed fd number) is silently closed.

This appears to be related to src/C++/HttpConnection.cpp (disconnect):

void HttpConnection::disconnect(int error) {
  if (error > 0) {
    send(HttpMessage::createResponse(error));
  }
  socket_close(m_socket);
  // m_socket is not invalidated here, so subsequent calls close the same fd again
}

Additionally, src/C++/HttpServer.cpp (onConnect) calls m_pServer->getMonitor().drop(s) after disconnect() already closed s, causing a 3rd close via SocketMonitor::drop().

Evidence: double-close confirmed via strace

The following strace output shows that the accepted fd is closed multiple times during a single HTTP request.

The test program starts HttpServer and sends one HTTP GET request.
No other threads are involved — this purely demonstrates the multiple-close behavior.

docker run --rm --cap-add=SYS_PTRACE qf-proof:latest \
  sh -c 'strace -f -tt -e trace=close,accept -o /tmp/st.log /app/proof 2>/dev/null; \
         echo "=== Accepted fd ==="; \
         grep accept /tmp/st.log; \
         ACCEPT_FD=$(grep -m1 "accept" /tmp/st.log | grep -oP "= \K[0-9]+"); \
         echo "=== close() calls on fd=${ACCEPT_FD} ==="; \
         grep "close(${ACCEPT_FD})" /tmp/st.log'

Result:

=== Accepted fd ===
11    02:11:36.664242 accept(5, NULL, NULL) = 7
=== close() calls on fd=7 ===
11    02:11:36.665034 close(7)          = 0
11    02:11:36.665464 close(7)          = -1 EBADF (Bad file descriptor)
11    02:11:36.665595 close(7)          = -1 EBADF (Bad file descriptor)

A single HTTP request causes close(7) to be called 3 times. The 1st succeeds, the 2nd and 3rd return EBADF (fd already closed).

Sources used in this section:

proof.cpp (test program source)
#include <quickfix/SessionSettings.h>
#include <quickfix/HttpServer.h>
#include <chrono>
#include <cstdio>
#include <cstring>
#include <sstream>
#include <thread>
#include <arpa/inet.h>
#include <csignal>
#include <netinet/in.h>
#include <sys/socket.h>
#include <unistd.h>

int main() {
    signal(SIGPIPE, SIG_IGN);
    constexpr int HTTP_PORT = 18080;

    // 1. Start HttpServer
    std::stringstream cfg;
    cfg << "[DEFAULT]\n";
    cfg << "HttpAcceptPort=" << HTTP_PORT << "\n";
    FIX::SessionSettings settings(cfg);
    FIX::HttpServer::startGlobal(settings);
    fprintf(stderr, "[PROOF] HttpServer started on port %d\n", HTTP_PORT);
    std::this_thread::sleep_for(std::chrono::milliseconds(200));

    // 2. Send a single HTTP GET request
    fprintf(stderr, "[PROOF] Sending 1 HTTP request...\n");
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    sockaddr_in a{};
    a.sin_family = AF_INET;
    a.sin_port = htons(HTTP_PORT);
    a.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
    if (connect(fd, reinterpret_cast<sockaddr*>(&a), sizeof(a)) == 0) {
        const char* req = "GET / HTTP/1.0\r\n\r\n";
        write(fd, req, strlen(req));
        char buf[256];
        // Read the full response
        while (read(fd, buf, sizeof(buf)) > 0) {}
    }
    close(fd);
    fprintf(stderr, "[PROOF] HTTP request done.\n");

    // Wait for HttpServer thread to finish processing
    std::this_thread::sleep_for(std::chrono::milliseconds(500));
    FIX::HttpServer::stopGlobal();
    return 0;
}
Dockerfile.proof
FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN dnf -y install gcc-c++ gcc git cmake make libxml2-devel libuuid-devel strace

# Requires: git clone https://github.com/quickfix/quickfix.git in the build context directory
COPY quickfix /opt/quickfix
COPY proof.cpp /app/proof.cpp

RUN cd /opt/quickfix && \
    cmake . -DCMAKE_BUILD_TYPE=Debug -DHAVE_MYSQL=OFF -DHAVE_SSL=OFF \
      -DHAVE_POSTGRESQL=OFF -DQUICKFIX_SHARED_LIBS=OFF \
      -DQUICKFIX_EXAMPLES=OFF -DQUICKFIX_TESTS=OFF && \
    cmake --build . -j$(nproc) && \
    echo "quickfix build OK"

RUN g++ -std=c++17 -O0 -g -pthread \
    -I /opt/quickfix/include \
    /app/proof.cpp /opt/quickfix/lib/libquickfix.a \
    -lxml2 -luuid -lpthread \
    -o /app/proof && \
    echo "proof built"

WORKDIR /app
Build command
docker build -f Dockerfile.proof -t qf-proof:latest .

Impact: fd theft in multithreaded applications

The double-close causes actual harm when other threads are present.
To demonstrate, another thread creates pipe() fds (completely unrelated to HttpServer) and checks if they remain valid.
If HttpServer's stale close hits one of those pipe fds, fcntl() returns EBADF — proving the fd was closed by something else.

What happens:

  1. HttpServer accepts a connection, gets fd N
  2. HttpServer calls close(N) — fd N is now free
  3. Another thread calls pipe() — OS assigns fd N to the pipe
  4. HttpServer calls close(N) again (bug) — destroys the pipe
  5. The other thread's pipe is now broken, even though it never closed it

Run command:

docker build -f Dockerfile.theft -t qf-theft:latest .
docker run --rm qf-theft:latest /app/test_fd_theft

Result:

Checked 1061323 pipe fds. Stolen: 9
FAIL: 9 pipe fd(s) were closed by HttpServer (fd theft)

9 pipe fds — completely unrelated to HttpServer — were closed by HttpServer's double-close.
These threads never called close() on their own pipe fds, yet the fds became invalid.

Sources used in this section:

test_fd_theft.cpp
// Demonstrates that HttpServer's double-close can destroy unrelated file descriptors.
//
// Thread A: Sends HTTP requests to HttpServer (triggers the double-close bug)
// Thread B: Creates pipe() fds and checks if they remain valid
// If HttpServer's stale close hits a pipe fd, fcntl() will return EBADF.
//
// A pipe fd has nothing to do with HttpServer — if it becomes invalid,
// the only explanation is that something else closed it.

#include <quickfix/SessionSettings.h>
#include <quickfix/HttpServer.h>

#include <atomic>
#include <chrono>
#include <cstdio>
#include <cstring>
#include <sstream>
#include <thread>

#include <arpa/inet.h>
#include <csignal>
#include <fcntl.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <unistd.h>

static std::atomic<bool> g_stop{false};
static std::atomic<long> g_stolen{0};
static std::atomic<long> g_checked{0};
static constexpr int HTTP_PORT = 18080;

static bool fd_alive(int fd) { return fcntl(fd, F_GETFD) != -1; }

// Thread A: Send HTTP requests to trigger HttpServer::onConnect -> double-close
static void http_hammer() {
    while (!g_stop.load()) {
        int fd = socket(AF_INET, SOCK_STREAM, 0);
        if (fd < 0) continue;
        sockaddr_in a{};
        a.sin_family = AF_INET;
        a.sin_port = htons(HTTP_PORT);
        a.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
        if (connect(fd, reinterpret_cast<sockaddr*>(&a), sizeof(a)) == 0) {
            write(fd, "GET / HTTP/1.0\r\n\r\n", 18);
            char buf[64];
            read(fd, buf, sizeof(buf));
        }
        close(fd);
    }
}

// Thread B: Create pipe fds and check if they get stolen
static void pipe_victim() {
    while (!g_stop.load()) {
        int pipefd[2];
        if (pipe(pipefd) != 0) continue;

        g_checked.fetch_add(1);

        // Brief hold — check if our pipe fd was closed by someone else
        for (int i = 0; i < 32; i++) {
            if (!fd_alive(pipefd[0]) || !fd_alive(pipefd[1])) {
                g_stolen.fetch_add(1);
                break;
            }
        }

        // Clean up (only if still valid)
        if (fd_alive(pipefd[0])) close(pipefd[0]);
        if (fd_alive(pipefd[1])) close(pipefd[1]);
    }
}

int main() {
    signal(SIGPIPE, SIG_IGN);

    // Start HttpServer
    std::stringstream cfg;
    cfg << "[DEFAULT]\nHttpAcceptPort=" << HTTP_PORT << "\n";
    FIX::SessionSettings settings(cfg);
    FIX::HttpServer::startGlobal(settings);
    std::this_thread::sleep_for(std::chrono::milliseconds(100));

    // Start threads
    std::thread hammers[4];
    for (auto& t : hammers) t = std::thread(http_hammer);

    std::thread victims[4];
    for (auto& t : victims) t = std::thread(pipe_victim);

    // Run for 8 seconds
    std::this_thread::sleep_for(std::chrono::seconds(8));
    g_stop.store(true);

    long stolen = g_stolen.load();
    long checked = g_checked.load();

    fprintf(stderr, "Checked %ld pipe fds. Stolen: %ld\n", checked, stolen);
    if (stolen > 0) {
        fprintf(stderr, "FAIL: %ld pipe fd(s) were closed by HttpServer (fd theft)\n", stolen);
        _exit(1);
    } else {
        fprintf(stderr, "PASS: no fd theft detected\n");
        _exit(0);
    }
}
Dockerfile.theft
FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN dnf -y install gcc-c++ gcc git cmake make libxml2-devel libuuid-devel

# Requires: git clone https://github.com/quickfix/quickfix.git in the build context directory
COPY quickfix /opt/quickfix
COPY test_fd_theft.cpp /app/test_fd_theft.cpp

# Build QuickFIX
RUN cd /opt/quickfix && \
    cmake . -DCMAKE_BUILD_TYPE=Debug -DHAVE_MYSQL=OFF -DHAVE_SSL=OFF \
      -DHAVE_POSTGRESQL=OFF -DQUICKFIX_SHARED_LIBS=OFF \
      -DQUICKFIX_EXAMPLES=OFF -DQUICKFIX_TESTS=OFF && \
    cmake --build . -j$(nproc) && \
    echo "quickfix build OK"

# Build test
RUN g++ -std=c++17 -O2 -g -pthread \
    -I /opt/quickfix/include \
    /app/test_fd_theft.cpp /opt/quickfix/lib/libquickfix.a \
    -lxml2 -luuid -lpthread \
    -o /app/test_fd_theft && \
    echo "test built"

WORKDIR /app

Possible Fix (suggestion)

As one possible approach, invalidating m_socket after close may mitigate the issue:

void HttpConnection::disconnect(int error) {
  if (!socket_isValid(m_socket)) return;  // already closed — do nothing
  if (error > 0) {
    send(HttpMessage::createResponse(error));
  }
  socket_close(m_socket);
  socket_invalidate(m_socket);            // set to -1 so subsequent calls are no-ops
}

Both socket_isValid() and socket_invalidate() already exist in Utility.h.

Note:
This addresses the double-close from disconnect() being called multiple times.
The drop(s) call in onConnect() uses a local variable s (not m_socket), so it may still cause an additional stale close.
A complete fix may require further changes to the close ownership between HttpConnection and SocketMonitor.

Environment (reproduced on)

  • QuickFIX version: 1.16.0 (commit 386ce46)
  • Operating system: Ubuntu 24.04.4 LTS (Docker container on EC2)
  • Compiler: g++, C++17
  • Architecture: x86_64

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions