Investigating the "Resource temporarily unavailable" Error on Nginx on Fargate
I was running Nginx on Fargate, but during testing I kept running into an issue where requests would clog up almost immediately.
Here is a summary of what I investigated at the time.
Setting the error log level to debug
It was originally set to info, but since I couldn’t find any relevant logs,
I set the error log level to debug in the nginx configuration file and checked the logs.
1 | error_log stderr debug; |
Then the following log showed up.
1 | accept() not ready (11: Resource temporarily unavailable) |
It says that accept() is not ready due to a temporary lack of resources.
Given how Nginx handles communication, the part that accept() is responsible for is easy to understand from the diagram below.
Looking into net.core.somaxconn
net.core.somaxconn is the maximum length of the queue that stores requests accepted by a TCP socket.
I came up with the hypothesis that the number of requests it can accept might be too small.
I entered the Nginx container and ran the following.
1 | sysctl net.core.somaxconn |
That’s very small!
I wanted to change this setting!
Trying to change it in the Dockerfile
1 | RUN sysctl -w net.core.somaxconn=1024 |
It errored out (T_T).
1 | #21 0.370 sysctl: error setting key 'net.core.somaxconn': Read-only file system |
Could it be done on Fargate?
1 | docker run --sysctl net.core.somaxconn=65535 ... |
Could it be done through Fargate settings?
【週刊 Ask An Expert #04】AWS Loft Tokyo で受けた質問まとめ #AWSLoft
Q: I want to change net.core.somaxconn on Fargate
At present, you can’t change it on Fargate, so please solve it by launching more tasks.
Conclusion
As things stand, the only way to resolve the Resource temporarily unavailable error that occurs with Nginx is to simply increase the number of tasks!
It has been raised as an issue on the AWS containers roadmap.
The day Fargate supports this will surely come eventually… I hope!
Investigating the "Resource temporarily unavailable" Error on Nginx on Fargate
https://kenzo0107.github.io/en/2021/04/16/nginx-on-fargate-somaxconn/