Commit 8c36d5b8 authored by Marc Gravell's avatar Marc Gravell

- try to auto-detect `LegacyAspNetSynchronizationContext` and enable the feature-flag

- additional guidance on thread-theft
- use UQUWI for continuations
- remove handling from simple (non-task) results; no risk of thread-theft
parent d37289b2
...@@ -45,7 +45,15 @@ configure ASP.NET with: ...@@ -45,7 +45,15 @@ configure ASP.NET with:
<add key="aspnet:UseTaskFriendlySynchronizationContext" value="false" /> <add key="aspnet:UseTaskFriendlySynchronizationContext" value="false" />
``` ```
In this scenario, we would once again end up with the reader being stolen and used for or
```
<httpRuntime targetFramework="4.5" />
```
([citation](https://devblogs.microsoft.com/aspnet/all-about-httpruntime-targetframework))
In these scenarios, we would once again end up with the reader being stolen and used for
processing your application logic. This can doom any further `await`s to timeouts, processing your application logic. This can doom any further `await`s to timeouts,
either temporarily (until the application logic chooses to release the thread), or permanently either temporarily (until the application logic chooses to release the thread), or permanently
(essentially deadlocking yourself). (essentially deadlocking yourself).
......
...@@ -3,6 +3,11 @@ ...@@ -3,6 +3,11 @@
Verify what's the maximum bandwidth supported on your client and on the server where redis-server is hosted. If there are requests that are getting bound by bandwidth, it will take longer for them to complete and thereby can cause timeouts. Verify what's the maximum bandwidth supported on your client and on the server where redis-server is hosted. If there are requests that are getting bound by bandwidth, it will take longer for them to complete and thereby can cause timeouts.
Similarly, verify you are not getting CPU bound on client or on the server box which would cause requests to be waiting for CPU time and thereby have timeouts. Similarly, verify you are not getting CPU bound on client or on the server box which would cause requests to be waiting for CPU time and thereby have timeouts.
Are you experiencing "thread theft" of the reader?
---
The parameter “`qs`” in the error message tells you the state of the reader; if this is frquently reporting `CompletePendingMessage`,
it is possible that the reader loop has been hijacked; see [Thread Theft](ThreadTheft) for specific guidance.
Are there commands taking long time to process on the redis-server? Are there commands taking long time to process on the redis-server?
--------------- ---------------
There can be commands that are taking long time to process on the redis-server causing the request to timeout. Few examples of long running commands are mget with large number of keys, keys * or poorly written lua script. You can run the SlowLog command to see if there are requests taking longer than expected. More details regarding the command can be found [here](https://redis.io/commands/slowlog). There can be commands that are taking long time to process on the redis-server causing the request to timeout. Few examples of long running commands are mget with large number of keys, keys * or poorly written lua script. You can run the SlowLog command to see if there are requests taking longer than expected. More details regarding the command can be found [here](https://redis.io/commands/slowlog).
......
...@@ -45,6 +45,18 @@ public static void SetFeatureFlag(string flag, bool enabled) ...@@ -45,6 +45,18 @@ public static void SetFeatureFlag(string flag, bool enabled)
} }
} }
static ConnectionMultiplexer()
{
bool value = false;
try
{ // attempt to detect a known problem scenario
value = SynchronizationContext.Current?.GetType()?.Name
== "LegacyAspNetSynchronizationContext";
}
catch { }
SetFeatureFlag(nameof(FeatureFlags.PreventThreadTheft), value);
}
/// <summary> /// <summary>
/// Returns the state of a feature flag; this should only be used under support guidance /// Returns the state of a feature flag; this should only be used under support guidance
/// </summary> /// </summary>
......
...@@ -27,15 +27,7 @@ internal abstract class SimpleResultBox : IResultBox ...@@ -27,15 +27,7 @@ internal abstract class SimpleResultBox : IResultBox
void IResultBox.SetException(Exception exception) => _exception = exception ?? CancelledException; void IResultBox.SetException(Exception exception) => _exception = exception ?? CancelledException;
void IResultBox.Cancel() => _exception = CancelledException; void IResultBox.Cancel() => _exception = CancelledException;
static readonly WaitCallback s_ActivateContinuations = state => ((SimpleResultBox)state).ActivateContinuationsImpl();
void IResultBox.ActivateContinuations() void IResultBox.ActivateContinuations()
{
if (ConnectionMultiplexer.PreventThreadTheft)
ThreadPool.QueueUserWorkItem(s_ActivateContinuations, this);
else
ActivateContinuationsImpl();
}
private void ActivateContinuationsImpl()
{ {
lock (this) lock (this)
{ // tell the waiting thread that we're done { // tell the waiting thread that we're done
...@@ -119,8 +111,8 @@ T IResultBox<T>.GetResult(out Exception ex, bool _) ...@@ -119,8 +111,8 @@ T IResultBox<T>.GetResult(out Exception ex, bool _)
static readonly WaitCallback s_ActivateContinuations = state => ((TaskResultBox<T>)state).ActivateContinuationsImpl(); static readonly WaitCallback s_ActivateContinuations = state => ((TaskResultBox<T>)state).ActivateContinuationsImpl();
void IResultBox.ActivateContinuations() void IResultBox.ActivateContinuations()
{ {
if (ConnectionMultiplexer.PreventThreadTheft) if ((Task.CreationOptions & TaskCreationOptions.RunContinuationsAsynchronously) == 0)
ThreadPool.QueueUserWorkItem(s_ActivateContinuations, this); ThreadPool.UnsafeQueueUserWorkItem(s_ActivateContinuations, this);
else else
ActivateContinuationsImpl(); ActivateContinuationsImpl();
} }
...@@ -145,15 +137,14 @@ private void ActivateContinuationsImpl() ...@@ -145,15 +137,14 @@ private void ActivateContinuationsImpl()
public static IResultBox<T> Create(out TaskCompletionSource<T> source, object asyncState) public static IResultBox<T> Create(out TaskCompletionSource<T> source, object asyncState)
{ {
// since 2.0, we only support platforms where this is correctly implemented
const TaskCreationOptions CreationOptions = TaskCreationOptions.RunContinuationsAsynchronously;
// it might look a little odd to return the same object as two different things, // it might look a little odd to return the same object as two different things,
// but that's because it is serving two purposes, and I want to make it clear // but that's because it is serving two purposes, and I want to make it clear
// how it is being used in those 2 different ways; also, the *fact* that they // how it is being used in those 2 different ways; also, the *fact* that they
// are the same underlying object is an implementation detail that the rest of // are the same underlying object is an implementation detail that the rest of
// the code doesn't need to know about // the code doesn't need to know about
var obj = new TaskResultBox<T>(asyncState, CreationOptions); var obj = new TaskResultBox<T>(asyncState, ConnectionMultiplexer.PreventThreadTheft
? TaskCreationOptions.None // if we don't trust the TPL/sync-context, avoid a double QUWI dispatch
: TaskCreationOptions.RunContinuationsAsynchronously);
source = obj; source = obj;
return obj; return obj;
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment