Last week I blogged about the integration of Akka.NET and ASP.NET Core. Today I would like to discuss possible problems you may face if you decide to deploy this kind of application, and how you can address them.

Where are all my books?

If you, for some unknown reason, decide to deploy the demo application from the previous article to IIS, you can observe some really strange behavior. All you have to do is play with the app a little after it starts (add a few books), and then let it rest for a while (twenty minutes should be enough). When you get back and send a request to fetch the list of books available in the system, you may be surprised. There are no books available.

I know we didn’t add persistence to our BooksManagerActor. But shouldn’t it preserve its internal state anyway. At least, until we explicitly decide to shut down the app. What happened, then?

IIS happened.

The tale of IIS and ASP.NET Core

Apparently, when you are hosting your app inside of IIS (Internet Information Services), the application pool your app lives in can be stopped and started at the whim of IIS. In other words, your ActorSystem can be stopped at any given time.

This is what happened with our app. After too much idle time¹, IIS thought it was a good idea to do some housekeeping and took our application down. Even though it was new and shiny ASP.NET Core app hosted by Kestrel with IIS configured as a reverse proxy.

It struck me really hard when I first saw this. After the introduction of Kestrel as a default web server, I was under impression that problems with IIS restarting my apps are gone. Boy was I wrong!

To understand this behavior we have to take a closer look at how ASP.NET Core apps actually work behind IIS. When you decide to deploy your app on Linux with Nginx or Apache, a reverse proxy responsibility is clear and well defined. All it does is redirect the HTTP requests to the ASP.NET Core app. When you, on the other hand, use Windows and IIS, the situation is slightly different. In order to provide several features, such as Windows Authentication (out of the box for ASP.NET Core applications) a special native IIS module was introduced: ASP.NET Core Module (ANCM). Its job is to plug into the IIS pipeline and redirect requests to backend ASP.NET Core apps. Besides that, it is also responsible for process management. If you wondered who kicks off your application when the first request arrives, who restarts it when it crashes, or finally, who kills it when IIS thinks it should be killed… this is your culprit!

The relationship between IIS, the ASP.NET Core Module, and ASP.NET Core apps.

Even though ASP.NET Core apps run in a process separate from the IIS worker process, what you end up with is basically the same behavior as seen back in the days when ASP.NET 4.x apps were in their prime.

Beyond IIS

From Akka perspective, this behavior is unacceptable. You shouldn’t have inherently stateful application, and in the same time let it loose its state at IIS wish. Even if the state can be recovered. It’s fundamentally wrong.

Is there anything we can do? Actually, there is. We can, for instance, lay off IIS and try to use Kestrel on its own.

If you do so, you may notice significant performance improvements. It’s because running ASP.NET Core behind IIS configured as a reverse proxy has a very high-performance overhead. Microsoft announced, that in ASP.NET Core 2.2 a new in-process hosting model will be available for ANCM. Initial results are very promising - over 4x better performance.

If, however, a reverse proxy server is required², you can try out some other technologies, for instance, Nginx. It is a really good, battle-tested piece of software, that should you live up to your expectations.

But what if using IIS is a necessity? I know in certain organizations it might be non-negotiable. Then you can tweak your IIS to never, ever, under any circumstances recycle your application. There is a great reply on StackOverflow describing how it can be achieved. If you follow it correctly, the IIS should let your application live without any interruptions. The downside of this solution is that the behavior of your system depends on the proper configuration of the IIS. Moreover, you will have to educate your IT operations that this particular service requires special treatment.

The last option (the preferred one) is to use a very lightweight³ ActorSystem inside your ASP.NET Core app, and offload actual work to a separate service via Akka Remoting. This new service could be a Windows Service, .NET Core Console App hosted by Docker or similar. The only requirement is that it must be free from all the IIS goodness. This way your precious state will be safe from IIS quirks and idiosyncrasies, and at the same time, ASP.NET Core will power your API layer. That’s a win-win scenario. Next time we will learn how this architecture can be implemented.

Footnotes: