SignalR hub running as an EKS pod handles 700 VUs via ALB but fails at 720+, yet successfully handles 1000 VUs via kubectl port-forward with 85% free CPU and 43% free memory. This confirms the application is healthy. I am new to AWS , I wanted to understand what might be the root cause this issue. I am performing a load test using k6 scripts.
Also, I am connected to Redis cache (AWS Elasticache) instance c6 large as backpane for scalability.
I am trying to understand is it server side resource exhaustion or ALB node scaling issue?
Error Logs-
INFO[0524] [VU 728] Starting sustained connection test - Target: https://xxxxxxxxxxxxxxxxxxx/hubs/notification source=console
INFO[0524] [VU 728] Will send messages every 15000ms for 20 minutes source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0524] [VU 728] Negotiation failed source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0524] [VU 728] Negotiation failed source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0524] [VU 728] Negotiation failed source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0524] [VU 728] Negotiation failed source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
INFO[0524] [VU 726] WebSocket opened source=console
ERRO[0524] [VU 728] Negotiation failed source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0524] [VU 728] Negotiation failed source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0524] [VU 728] Negotiation failed source=console
WARN[0524] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0524] [VU 728] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0524] [VU 728] Negotiation failed source=console
INFO[0524] [VU 726] Handshake complete in 1067ms source=console
INFO[0525] [VU 729] Starting sustained connection test - Target: https://xxxxxxxxxxxxxxxxxxx/hubs/notification source=console
INFO[0525] [VU 729] Will send messages every 15000ms for 20 minutes source=console
WARN[0525] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0525] [VU 729] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0525] [VU 729] Negotiation failed source=console
INFO[0525] [VU 727] WebSocket opened source=console
WARN[0525] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0525] [VU 729] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0525] [VU 729] Negotiation failed source=console
WARN[0525] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
ERRO[0525] [VU 729] NEGOTIATION_HTTP_ERROR: HTTP error during negotiation source=console
ERRO[0525] [VU 729] Negotiation failed source=console
WARN[0525] Request Failed error="Post "https://xxxxxxxxxxxxxxxxxxx/hubs/notification/negotiate\": EOF"
SignalR configuration:-
public static IServiceCollection AddSignalRServices(this IServiceCollection services, IConfiguration configuration, IWebHostEnvironment environment)
{
var signalRBuilder = services.AddSignalR(options =>
{
options.ClientTimeoutInterval = TimeSpan.FromMinutes(2);
options.KeepAliveInterval = TimeSpan.FromSeconds(
configuration.GetValue<int>("SignalR_KeepAliveSeconds", 30));
options.HandshakeTimeout = TimeSpan.FromSeconds(
configuration.GetValue<int>("SignalR_HandshakeTimeoutSeconds", 15));
options.MaximumParallelInvocationsPerClient =
configuration.GetValue<int>("SignalR_MaxParallelInvocations", 1);
options.MaximumReceiveMessageSize =
configuration.GetValue<long?>("SignalR_MaxMessageSize", 32 * 1024);
options.StreamBufferCapacity =
configuration.GetValue<int>("SignalR_StreamBufferCapacity", 10);
// Adjusted settings for Debugging and performance tuning
options.MaximumReceiveMessageSize = 32768;
options.StreamBufferCapacity = 5; // Reduced from 10
// Timeouts - balance connection count vs responsiveness
options.KeepAliveInterval = TimeSpan.FromSeconds(20);
options.HandshakeTimeout = TimeSpan.FromSeconds(20);
// Limit parallel invocations to reduce CPU load
options.MaximumParallelInvocationsPerClient = 1;
options.EnableDetailedErrors = false; // Disable in production
});
signalRBuilder.AddMessagePackProtocol();
services.AddRedisBackplane(signalRBuilder, configuration, environment);
return services;
}
private static void AddRedisBackplane(
this IServiceCollection services,
ISignalRServerBuilder signalRBuilder,
IConfiguration configuration,
IWebHostEnvironment environment)
{
var redisConfig = configuration.GetRedisConfiguration();
var redisHost = redisConfig.Host;
var redisPort = redisConfig.Port;
var redisPassword = redisConfig.Password;
if (string.IsNullOrEmpty(redisHost))
{
Console.WriteLine("Redis backplane is enabled but Redis host is not configured.");
return;
}
var useSSL = configuration.GetValue<bool>("RedisConfig_UseSsl", true);
var redisConnectionString = BuildRedisConnectionString(redisHost, redisPort, redisPassword, useSSL);
signalRBuilder.AddStackExchangeRedis(redisConnectionString, options =>
{
options.Configuration.ChannelPrefix = StackExchange.Redis.RedisChannel.Literal("SignalR");
options.Configuration.AbortOnConnectFail = false;
options.Configuration.ConnectTimeout = 15000;
options.Configuration.SyncTimeout = 30000;
options.Configuration.AsyncTimeout = 30000;
options.Configuration.KeepAlive = 60;
options.Configuration.ConnectRetry = 3;
options.Configuration.ReconnectRetryPolicy = new StackExchange.Redis.ExponentialRetry(5000);
options.Configuration.DefaultDatabase = 0;
options.Configuration.Proxy = StackExchange.Redis.Proxy.None;
});
}
Kestrel Configuration:-
ThreadPool.SetMinThreads(200, 200);
Console.WriteLine("[ThreadPool] Min threads set to 200 for high concurrency (SignalR 1000+ connections)");
builder.WebHost.ConfigureKestrel(options =>
{
options.Limits.KeepAliveTimeout = TimeSpan.FromSeconds(280);
options.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(60);
options.Limits.Http2.KeepAlivePingDelay = TimeSpan.FromSeconds(10);
options.Limits.MaxConcurrentConnections = 5100;
options.Limits.MaxConcurrentUpgradedConnections = 5100;
options.Limits.MinRequestBodyDataRate = null;
options.Limits.MinResponseDataRate = null;
options.Limits.RequestHeadersTimeout = TimeSpan.FromSeconds(30);
options.Limits.MinRequestBodyDataRate = null;
options.Limits.MinResponseDataRate = null;
options.Limits.MaxRequestBodySize = 10 * 1024;
options.Limits.MaxRequestBufferSize = 1024 * 1024;
options.Limits.MaxRequestHeadersTotalSize = 32 * 1024;
});
