Some of our tests were randomly hanging on the Linux machine. This was a cause for concern, obviously, so we started investigating that.

The problem was that we couldn’t really reproduce this except by running our full code, in which case it would hang. It clearly seemed timing related, because it wouldn’t always hang, but it was pretty consistent.  We resorted to sprinkling printf throughout our code to figure out what was going on, and we found that a particular request would get to the server, the server would respond to it, and then the client would hang.

Looking at TCP traces, we could clearly see that the response was generated by the server and sent to the client, but something was funky there. We reported it and try to figure out what was going on in the meantime.

We finally managed to get an isolated reproduction:

As a side note, the response from the Core FX & Kestrel guys is absolutely amazing. In a short while, it was determined to be an issue in Kestrel and fixed.

The details of the issue are really interesting.

1. The client (libcurl) starts sending the request.
2. Kestrel sends an error response.
3. The client sees the error response and stops sending the request (this is a libcurl behavior), but does NOT close the sending side of the connection (HTTP clients typically wait until they receive the entire response before closing the connection, because most HTTP servers treat a mid-request FIN like a connection reset).
4. After the application runs, Kestrel tries to consume the rest of the request body from the client. But the client has stopped sending the request, so Kestrel hangs here.
5. The client hangs waiting for the chunked terminator from the server, which is only sent after it consumes the rest of the request body (we'll work on improving this behavior).

This behavior is specific to libcurl,  the WinHTTP implementation sends the full body of the request before processing any of the response.

As an aside, this difference in behavior is actually pretty interesting to know, because we need those low level details when designing interactions. For example, if I send a long request to the server, and the server want to tell me that it is not going to accept it, being able to know about it early is very advantageous.