In an Azure function transferring data from a MongoDB to an SQL-Server we were seeing connection errors on the MongoDB-side:
MongoConnectionException: An exception occurred while opening a connection to the server.
These occurred suddenly after unpredictable times during the duration of a function run, and they occurred only on Azure, not when running the function locally.
When further investigating the exception being thrown inside the function, we discovered both that it was caused by a broken TCP-socket and that it always happened when accessing the secondary MongoDB server:
MongoConnectionException: An exception occurred while opening a connection to the server.
SocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
{ ServerId : { ClusterId : 1, EndPoint : „Unspecified/xxx-xxx-xxx.mongodb.net:27017“ } }
Other developers have discovered (and Microsoft has documented) that Azure VM’s have a very short socket idle time of 4 minutes by default:
https://serverfault.com/questions/661704/azure-closing-idle-network-connections
https://azure.microsoft.com/en-us/blog/new-configurable-idle-timeout-for-azure-load-balancer/
Seemingly this is true for the Azure function host as well. In our case the function was usually querying the primary MongoDB server, but the secondary only sporadically.
The C#/.NET-MongoDB-Driver keeps connections to the Mongo-servers in a connection pool, and we had a large chance of trying to re-use a connection to the secondary server whose underlying socket had already been closed by the function host.
The C#-driver has no built-in option to send keep-alive messages. Therefore, we solved the problem by setting the maximum connection idle time for the connection pool to 3 minutes:
mongoSettings.MaxConnectionIdleTime = new TimeSpan(0, 3, 0);
Now the driver would no longer try to revive connections that had been idle for more than 3 minutes, but instead reconnect. After deploying the modified app, said connection errors disappeared.
The issue has been addressed by the MongoDB team and a similar solution has been suggested, but not in direct reference to the C# driver: