franklixuefei Github contribution chart
franklixuefei Github Stats
franklixuefei Most Used Languages

Activity

23 Sep 2022

Issue Comment

Franklixuefei

Calling CancellationTokenSource.Cancel() upon yarp config change results in stack overflow

Describe the bug

A clear and concise description of what the bug is.

We implemented our custom IProxyConfig as follows:

 internal class MoxyDynamicProxyConfig : IProxyConfig
    {
        private readonly MoxyConfigMonitor<InternalYarpConfig> configMonitor;

        public MoxyDynamicProxyConfig(IOptionsMonitor<MoxyOptions> optionsMonitor)
        {
            this.logger = logger;
            this.configMonitor = new MoxyConfigMonitor(optionsMonitor);
        }

        public IReadOnlyList<RouteConfig> Routes => ...

        public IReadOnlyList<ClusterConfig> Clusters => ...

        public IChangeToken ChangeToken => this.configMonitor.CancellationChangeToken; 

where MoxyConfigMonitor is as follows:

 public class MoxyConfigMonitor
    {
        private readonly IOptionsMonitor<MoxyOptions> optionsMonitor;

        private CancellationTokenSource cts;

        private IChangeToken cancellationChangeToken;

        public MoxyConfigMonitor(IOptionsMonitor<MoxyOptions> optionsMonitor)
        {
            this.cts = new CancellationTokenSource();
            this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token);
            this.optionsMonitor = optionsMonitor;
            if (this.TryLoadOptions(this.optionsMonitor.CurrentValue, out var current))
            {
                this.Current = current;
            }

            this.optionsMonitor.OnChange((newOptions, name) =>
            {
                if (this.TryLoadOptions(newOptions, out var current))
                {
                    this.Current = current;
                    this.cts.Cancel(); // <-- this triggers the infinite event-trigger loop causing stack overflow
                    this.cts.Dispose();
                    this.cts = new CancellationTokenSource();
                    this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token);
                }
            });
        }

        public IChangeToken CancellationChangeToken => this.cancellationChangeToken; 

Whenever there's a change to our Yarp config, this.optionsMonitor.OnChange() will be triggered and subsequently, this.cts.Cancel() will be triggered. So far so good.

However, right after the Cancel() call, our console shows the following logs:

... at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.g__SignalChange|31_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Azure.Moxy.Configuration.MoxyConfigMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__5_0(Microsoft.Azure.Moxy.DataContracts.MoxyOptions, System.String) at Microsoft.Extensions.Options.OptionsMonitor1+ChangeTrackerDisposable[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChange(System.__Canon, System.String) at Microsoft.Extensions.Options.OptionsMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InvokeChanged(System.String) at Microsoft.Extensions.Options.OptionsMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__6_2(System.String) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.Configuration.ConfigurationRoot.RaiseChanged() at Microsoft.Extensions.Configuration.ConfigurationRoot.<.ctor>b__3_1() at Microsoft.Extensions.Primitives.ChangeToken+<>c.b__0_0(System.Action) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.Configuration.ConfigurationProvider.OnReload() at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean) at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_1() at Microsoft.Extensions.Primitives.ChangeToken+<>c.b__0_0(System.Action) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+<>c.<.cctor>b__43_0(System.Object) at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() at System.Threading.Thread.StartCallback()

(The bold text is the stack trace that got repeated indefinitely)

We also set a breakpoint at the GetConfig() method under our custom IProxyConfigProvider, and GetConfig() was also called for infinitely many times.

To Reproduce

Please see above.

Further technical details

  • Include the version of the packages you are using
  • The platform (Linux/macOS/Windows)

Forked On 23 Sep 2022 at 05:36:56

Franklixuefei

Thanks @Tratcher !! Spot on!

Commented On 23 Sep 2022 at 05:36:56
Issue Comment

Franklixuefei

Calling CancellationTokenSource.Cancel() upon yarp config change results in stack overflow

Describe the bug

A clear and concise description of what the bug is.

We implemented our custom IProxyConfig as follows:

 internal class MoxyDynamicProxyConfig : IProxyConfig
    {
        private readonly MoxyConfigMonitor<InternalYarpConfig> configMonitor;

        public MoxyDynamicProxyConfig(IOptionsMonitor<MoxyOptions> optionsMonitor)
        {
            this.logger = logger;
            this.configMonitor = new MoxyConfigMonitor(optionsMonitor);
        }

        public IReadOnlyList<RouteConfig> Routes => ...

        public IReadOnlyList<ClusterConfig> Clusters => ...

        public IChangeToken ChangeToken => this.configMonitor.CancellationChangeToken; 

where MoxyConfigMonitor is as follows:

 public class MoxyConfigMonitor
    {
        private readonly IOptionsMonitor<MoxyOptions> optionsMonitor;

        private CancellationTokenSource cts;

        private IChangeToken cancellationChangeToken;

        public MoxyConfigMonitor(IOptionsMonitor<MoxyOptions> optionsMonitor)
        {
            this.cts = new CancellationTokenSource();
            this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token);
            this.optionsMonitor = optionsMonitor;
            if (this.TryLoadOptions(this.optionsMonitor.CurrentValue, out var current))
            {
                this.Current = current;
            }

            this.optionsMonitor.OnChange((newOptions, name) =>
            {
                if (this.TryLoadOptions(newOptions, out var current))
                {
                    this.Current = current;
                    this.cts.Cancel(); // <-- this triggers the infinite event-trigger loop causing stack overflow
                    this.cts.Dispose();
                    this.cts = new CancellationTokenSource();
                    this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token);
                }
            });
        }

        public IChangeToken CancellationChangeToken => this.cancellationChangeToken; 

Whenever there's a change to our Yarp config, this.optionsMonitor.OnChange() will be triggered and subsequently, this.cts.Cancel() will be triggered. So far so good.

However, right after the Cancel() call, our console shows the following logs:

... at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.g__SignalChange|31_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Azure.Moxy.Configuration.MoxyConfigMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__5_0(Microsoft.Azure.Moxy.DataContracts.MoxyOptions, System.String) at Microsoft.Extensions.Options.OptionsMonitor1+ChangeTrackerDisposable[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChange(System.__Canon, System.String) at Microsoft.Extensions.Options.OptionsMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InvokeChanged(System.String) at Microsoft.Extensions.Options.OptionsMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__6_2(System.String) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.Configuration.ConfigurationRoot.RaiseChanged() at Microsoft.Extensions.Configuration.ConfigurationRoot.<.ctor>b__3_1() at Microsoft.Extensions.Primitives.ChangeToken+<>c.b__0_0(System.Action) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.Configuration.ConfigurationProvider.OnReload() at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean) at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_1() at Microsoft.Extensions.Primitives.ChangeToken+<>c.b__0_0(System.Action) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+<>c.<.cctor>b__43_0(System.Object) at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() at System.Threading.Thread.StartCallback()

(The bold text is the stack trace that got repeated indefinitely)

We also set a breakpoint at the GetConfig() method under our custom IProxyConfigProvider, and GetConfig() was also called for infinitely many times.

To Reproduce

Please see above.

Further technical details

  • Include the version of the packages you are using
  • The platform (Linux/macOS/Windows)

Forked On 21 Sep 2022 at 12:23:18

Franklixuefei

The following code is what we had originally, and it works for previous version of Yarp (prior to the support of multiple config sources)

this.optionsMonitor.OnChange((newOptions, name) =>
{
    if (this.TryLoadOptions(newOptions, out var current))
    {
        this.Current = current;
        var oldCts = this.cts;
        this.cts = new CancellationTokenSource();
        this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token);

        oldCts.Cancel();
    }
}); 

this does not produce the infinite even triggering, but endpoints are not updated. I checked out the source code, and looks like the following code prevent that from happening. The following code piece was introduced along the "multiple config source" feature.

if (instance.LatestConfig.ChangeToken.HasChanged)
{
    var config = instance.Provider.GetConfig();
    ValidateConfigProperties(config);
    instance.LatestConfig = config;
    instance.LoadFailed = false;
    sourcesChanged = true;
} 

I think this is because by the time oldCts.Cancel(); was called, this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token); had been executed, resulting in a new CancellationChangeToken instance with HasChanged flag being false.

Commented On 21 Sep 2022 at 12:23:18
Issue Comment

Franklixuefei

Calling CancellationTokenSource.Cancel() upon yarp config change results in stack overflow

Describe the bug

A clear and concise description of what the bug is.

We implemented our custom IProxyConfig as follows:

 internal class MoxyDynamicProxyConfig : IProxyConfig
    {
        private readonly MoxyConfigMonitor<InternalYarpConfig> configMonitor;

        public MoxyDynamicProxyConfig(IOptionsMonitor<MoxyOptions> optionsMonitor)
        {
            this.logger = logger;
            this.configMonitor = new MoxyConfigMonitor(optionsMonitor);
        }

        public IReadOnlyList<RouteConfig> Routes => ...

        public IReadOnlyList<ClusterConfig> Clusters => ...

        public IChangeToken ChangeToken => this.configMonitor.CancellationChangeToken; 

where MoxyConfigMonitor is as follows:

 public class MoxyConfigMonitor
    {
        private readonly IOptionsMonitor<MoxyOptions> optionsMonitor;

        private CancellationTokenSource cts;

        private IChangeToken cancellationChangeToken;

        public MoxyConfigMonitor(IOptionsMonitor<MoxyOptions> optionsMonitor)
        {
            this.cts = new CancellationTokenSource();
            this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token);
            this.optionsMonitor = optionsMonitor;
            if (this.TryLoadOptions(this.optionsMonitor.CurrentValue, out var current))
            {
                this.Current = current;
            }

            this.optionsMonitor.OnChange((newOptions, name) =>
            {
                if (this.TryLoadOptions(newOptions, out var current))
                {
                    this.Current = current;
                    this.cts.Cancel(); // <-- this triggers the infinite event-trigger loop causing stack overflow
                    this.cts.Dispose();
                    this.cts = new CancellationTokenSource();
                    this.cancellationChangeToken = new CancellationChangeToken(this.cts.Token);
                }
            });
        }

        public IChangeToken CancellationChangeToken => this.cancellationChangeToken; 

Whenever there's a change to our Yarp config, this.optionsMonitor.OnChange() will be triggered and subsequently, this.cts.Cancel() will be triggered. So far so good.

However, right after the Cancel() call, our console shows the following logs:

... at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource.Register(System.Delegate, System.Object, System.Threading.SynchronizationContext, System.Threading.ExecutionContext) at System.Threading.CancellationToken.Register(System.Delegate, System.Object, Boolean, Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.ListenForConfigChanges() at Yarp.ReverseProxy.Management.ProxyConfigManager+d__29.MoveNext() at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Yarp.ReverseProxy.Management.ProxyConfigManager+d__29, Yarp.ReverseProxy, Version=1.1.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](d__29 ByRef) at Yarp.ReverseProxy.Management.ProxyConfigManager.ReloadConfigAsync() at Yarp.ReverseProxy.Management.ProxyConfigManager.g__ReloadConfig|31_1(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at Yarp.ReverseProxy.Management.ProxyConfigManager.g__SignalChange|31_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Azure.Moxy.Configuration.MoxyConfigMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__5_0(Microsoft.Azure.Moxy.DataContracts.MoxyOptions, System.String) at Microsoft.Extensions.Options.OptionsMonitor1+ChangeTrackerDisposable[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChange(System.__Canon, System.String) at Microsoft.Extensions.Options.OptionsMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InvokeChanged(System.String) at Microsoft.Extensions.Options.OptionsMonitor1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].<.ctor>b__6_2(System.String) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.Configuration.ConfigurationRoot.RaiseChanged() at Microsoft.Extensions.Configuration.ConfigurationRoot.<.ctor>b__3_1() at Microsoft.Extensions.Primitives.ChangeToken+<>c.b__0_0(System.Action) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode+<>c.b__9_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.Configuration.ConfigurationProvider.OnReload() at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean) at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_1() at Microsoft.Extensions.Primitives.ChangeToken+<>c.b__0_0(System.Action) at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].OnChangeTokenFired() at Microsoft.Extensions.Primitives.ChangeToken+ChangeTokenRegistration1+<>c[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].b__7_0(System.Object) at System.Threading.CancellationTokenSource.Invoke(System.Delegate, System.Object, System.Threading.CancellationTokenSource) at System.Threading.CancellationTokenSource+CallbackNode.ExecuteCallback() at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean) at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean) at System.Threading.CancellationTokenSource.Cancel() at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher+<>c.<.cctor>b__43_0(System.Object) at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() at System.Threading.Thread.StartCallback()

(The bold text is the stack trace that got repeated indefinitely)

We also set a breakpoint at the GetConfig() method under our custom IProxyConfigProvider, and GetConfig() was also called for infinitely many times.

To Reproduce

Please see above.

Further technical details

  • Include the version of the packages you are using
  • The platform (Linux/macOS/Windows)

Forked On 21 Sep 2022 at 12:15:32

Franklixuefei

@Tratcher Any help/suggestions would be greatly appreciated!

Commented On 21 Sep 2022 at 12:15:32