The original rules were stateful, meaning pf kept track of the state of
the traffic. That type of filtering is much more efficient. pf keeps
track of connections using port numbers. What was happening was the
client was re-using source port numbers before pf cleared that state.
When a new connection came in, pf had a state already from before (they
take 90s to be cleared), and the packet was seen as a state mismatch.
The reason it wasn’t timing out due to drops is that pf handles this
situation differently than other state-mismatch. Instead of dropping
the packet, it returns a reset. That is why the connections would fail
immediately.

You can see in the PF code here that RSTs are sent for state mismatch during a handshake. It’s around line 4517.