This is a topic of my online course on Udemy “Grafana and Graphite: Visualizing metrics for DevOps”. You can see the complete course here:
If you are building public facing web sites one of the things you want to achieve at the end of the project is a good performance under the load for your web site. That means, you have to make sure that your product works under a heavy load (e.g. 50 concurrent users, or 200 users per second etc.) even though at the moment you don’t think you would have that much load. Chances are that the web site attracts more and more users over time and then if it’s not a load tolerant web site it will start flaking, leaving you with an unhappy customer and ruined reputation.
There are many articles on the Internet about improving the performance of ASP.NET web sites, which all make sense; however, I think there are some more things you can do to save yourself from facing massive dramas. So what steps can be taken to produce a high-performance ASP.NET or ASP.NET MVC application?
Load test your application from early stages
Majority of developers tend to leave performing the load test (if they ever do it) to when the application is developed and has passed the integration and regression tests. Even though performing a load test at the end of the development process is better than not doing it at all, it might be way too late to fix the performance issues once your code has been already been written. A very common example of this issue is that when the application does not respond properly under load, scaling out (adding more servers) is considered. Sometimes this is not possible simply because the code is not suitable for achieving it. Like when the objects that are stored in Session are not serializable, and so adding more web nodes or more worker processes are impossible. If you find out that your application may require to be deployed on more than one server at the early stages of development, you will do your tests in an environment which is close to your final environment in terms of configuration and number of servers etc., then your code will be adapted a lot easier.
Use the high-performance libraries
Recently I was diagnosing the performance issues of a web site and I came across a hot spot in the code where JSON messages coming from a third-party web service had to be serialized several times. Those JSON messages were de-serialized by Newtonsoft.Json and tuned out that Newtonsoft.Json was not the fastest library when it came to de-serialization. Then we replaced Json.Net with a faster library (e.g. ServiceStack) and got a much better result.
Again if the load test was done at an early stage when we picked Json.Net as our serialization library we would have find that performance issue a lot sooner and would not have to make so many changes in the code, and would not have to re-test it entirely again.
Is your application CPU-intensive or IO-intensive?
Before you start implementing your web site and when the project is designed, one thing you should think about is whether your site is a CPU-intensive or IO-intensive? This is important to know your strategy of scaling your product.
For example if your application is CPU-intensive you may want to use a synchronous pattern, parallel processing and so forth whereas for a product that has many IO-bound operations such as communicating with external web services or network resources (e.g. a database) Task-based asynchronous pattern might be more helpful to scale out your product. Plus you may want to have a centralized caching system in place which will let you create Web Gardens and Web Farms in future, thus spanning the load across multiple worker processes or serves.
Use Task-based Asynchronous Model, but with care!
If your product relies many IO-bound operations, or includes long-running operations which may make the expensive IIS threads wait for an operation to complete, you better think of using the Task-based Asynchronous Pattern for your ASP.NET MVC project.
There are many tutorials on the Internet about asynchronous ASP.NET MVC actions (like this one) so in this blog post I refrain from explaining it. However, I just have to point out that traditional synchronous Actions in an ASP.NET (MVC) site keep the IIS threads busy until your operation is done or the request is processed. This means that if the site is waiting for an external resource (e.g. web service) to respond, the thread will be busy. The number of threads in .NET’s thread pool that can be used to process the requests are limited too, therefore, it’s important to release the threads as soon as possible. A task-based asynchronous action or method releases the thread until the request is processed, then grabs a new thread from the thread pool and uses it to return the result of the action. This way, many requests can be processed by few threads, which will lead to better responsiveness for your application.
Although task-based asynchronous pattern can be very handy for the right applications, it must be used with care. There are a few of concerns that you must have when you design or implement a project based on Task-based Asynchronous Pattern (TAP). You can see many of them in here, however, the biggest challenge that developers may face when using async and await keywords is to know that in this context they have to deal with threads slightly differently. For example, you can create a method that returns a Task (e.g. Task<Product>). Normally you can call .Run() method on that task or you can merely call task.Result to force running the task and then fetching the result. In a method or action which is built based on TBP, any of those calls will block your running thread, and will make your program sluggish or even may cause dead-locks.
Distribute caching and session state
It’s very common that developers build a web application on a single development machine and assume that the product will be running on a single server too, whereas it’s not usually the case for big public facing web sites. They often get deployed to more than one server which are behind a load balancer. Even though you can still deploy a web site with In-Proc caching on multiple servers using sticky session (where the load balancer directs all requests that belong to the same session to a single server), you may have to keep multiple copies of session data and cached data. For example if you deploy your product on a web farm made of four servers and you keep the session data in-proc, when a request comes through the chance of hitting a server that already contains a cached data is 1 in 4 or 25%, whereas if you use a centralized caching mechanism in place, the chance of finding a cached item for every request if 100%. This is crucial for web sites that heavily rely on cached data.
Another advantage of having a centralized caching mechanism (using something like App Fabric or Redis) is the ability to implement a proactive caching system around the actual product. A proactive caching mechanism may be used to pre-load the most popular items into the cache before they are even requested by a client. This may help with massively improving the performance of a big data driven application, if you manage to keep the cache synchronized with the actual data source.
Create Web Gardens
As it was mentioned before, in an IO-bound web application that involves quite a few long-running operations (e.g. web service calls) you may want to free up your main thread as much as possible. By default every web site is run under one main thread which is responsible to keep your web site alive, and unfortunately when it’s too busy, your site becomes unresponsive. There is one way of adding more “main threads” to your application which is achievable by adding more worker processes to your site under IIS. Each worker process will include a separate main thread therefore if one is busy there will be another one to process the upcoming processes.
Having more than one worker process will turn your site to a Web Garden, which requires your Session and Application data be persisted out-proc (e.g. on a state server or Sql Server).
Use caching and lazy loading in a smart way
There is no need to emphasize that if you cache a commonly accessed bit of data in memory you will be able to reduce the database and web service calls. This will specifically help with IO-bound applications that as I said before, may cause a lot of grief when the site is under load.
Another approach for improving the responsiveness of your site is using Lazy Loading. Lazy Loading means that an application does not have a certain piece of data, but it knows that where is that data. For example if there is a drop-down control on your web page which is meant to display list of products, you don’t have to load all products from the database once the page is loaded. You can add a jQuery function to your page which can populate the drop-down list the first time it’s pulled down. You can also apply the same technique in many places in your code, such as when you work with Linq queries and CLR collections.
Do not put C# code in your MVC views
Your ASP.NET MVC views get compiled at run time and not at compile time. Therefore if you include too much C# code in them, your code will not be compiled and placed in DLL files. Not only this will damage the testability of your software but also it will make your site slower because every view will take longer to get display (because they must be compiled). Another down side of adding code to the views is that they cannot be run asynchronously and so if you decide to build your site based on Task-based Asynchronous Pattern (TAP), you won’t be able to take advantage of asynchronous methods and actions in the views.
For example if there is a method like this in your code:
public async Task<string> GetName(int code)
var result = …
return await result;
This method can be run asynchronously in the context of an asynchronous ASP.NET MVC action like this:
public Task<ActionResult> Index(CancellationToken ctx)
var name = await GetName(100);
But if you call this method in a view, because the view is not asynchronous you will have to run it in a thread-blocking way like this:
var name = GetName(100).Result;
.Result will block the running thread until GetName() processes our request and so the execution of the app will halt for a while, whereas when this code is called using await keyword the thread is not blocked.
Use Fire & Forget when applicable
If two or more operations are not forming a single transaction you probably do not have to run them sequentially. For example if users can sign-up and create an account in your web site, and once they register you save their details in the database and then you send them an email, you don’t have to wait for the email to be sent to finalize the operation.
In such a case the best way of doing so is probably starting a new thread and making it send the email to the user and just get back to the main thread. This is called a fire and forgets mechanism which can improve the responsiveness of an application.
Build for x64 CPU
32-bit applications are limited to a lower amount of memory and have access to fewer calculation features/instructions of the CPU. To overcome these limitations, if your server is a 64-bit one, make sure your site is running under 64-bit mode (by making sure the option for running a site under 32-bit mode in IIS is not enabled). Then compile and build your code for x64 CPU rather than Any CPU.
One example of x64 being helpful is that to improve the responsiveness and performance of a data-driven application, having a good caching mechanism in place is a must. In-proc caching is a memory consuming option because everything is stored in the memory boundaries of the site’s application pool. For a x86 process, the amount of memory that can be allocated is limited to 4 GB and so if loads of data be added to the cache, soon this limit will be met. If the same site is built explicitly for a x64 CPU, this memory limit will be removed and so more items can be added to the cache thus less communication with the database which leads to a better performance.
Use monitoring and diagnostic tools on the server
There might be many performance issues that you never see them by naked eyes because they never appear in error logs. Identifying performance issues are even more daunting when the application is already on the production servers where you have almost no chance of debugging.
To find out the slow processes, thread blocks, hangs, and errors and so forth it’s highly recommended to install a monitoring and/or diagnostic tool on the server and get them to track and monitor your application constantly. I personally have used NewRelic (which is a SAS) to check the health of our online sites. See HERE for more details and for creating your free account.
Profile your running application
Once you finish the development of your site, deploy it to IIS, and then attach a profiler (e.g. Visual Studio Profiler) and take snapshots of various parts of the application. For example take a snapshot of purchase operation or user sign-up operation etc. Then check and see if there is any slow or blocking code there. Finding those hot spots at early stages might save you a great amount of time, reputation and money.
I haven’t written here for a while because I was involved in a job not related directly to ASP.NET development. In fact, I was consulting a small company to build up a software product line based on a standard software development process (i.e. MSF) and industry best practices.
The job urged me to write something about how to build up a small software product line or how to work on a software project following standard guidelines. It may be hard to believe, but I personally have seen a lot of companies, in various sizes, that do not have any strategy for their configuration management, requirement management and so on. Especially, small businesses suffer from this issue a lot but most of them frequently change their developers because the manager believes that developers do not work well! Some other ones know about the importance of having a well defined software development process but believe that they can’t afford the cost of hiring consultants or purchasing the tools. This post will show you up that software development can be organized perfectly with minimum cost and a little effort.
Begin with Standards
Whether you are going to launch a new project or you are maintaining an old application, you need some standards for coding conventions, database naming, coding principals, testing and so on. If you do not have any standards, the code your developers write will be very hard to understand and hard to maintain by other ones. Naming standards also can help you develop some automation tools because you can assume that some conditions are always met. For example, you can always be sure that the primary key field of each table is named “ID”. Standards should be granular, clear and short as much as possible. A very long and dense standard document is very hard to read, memorize and understand. For example, create some standard documents for the following subjects:
- C# (or other language that you use) naming conventions
- Database objects naming conventions
- UI standards
- C# design and coding standards and guidelines
- Database design standards and guidelines
It is very important you supervise that standards are always followed by team members. There are several tools that you can use to check if standards are correctly used. For instance, StyleCop can integrate with Visual Studio and analyze source code for specific conditions. See this link for more info: http://http://code.msdn.microsoft.com/sourceanalysis
It is highly recommended to create coding and other standards with cooperation of all team members. There are also several web resources about coding and design standards that you can refer to. Something that I personally did was to download and study some sample source code from Microsoft website and checked to see what kind of naming conventions have been used in them.
Setup a domain for your office!
It is recommended to have a domain-based internal network for your office. Without a domain you can not control the software or servers you setup. Big organizations often setup a domain, or a secondary domain, for their software development section.
Setup an internal mail server
The office I told you about, the one which I worked on the promoting their software development process, used to exchange their internal documents or messages thorough external email accounts (i.e. Gmail) . There is no point in sending internal valuable information included in internal documents out of the company! Therefore, it is much better to set up an internal email server, i.e. Microsoft Exchange Server, and use it for internal communications. If you don’t have money (!) , setup a free pop3 mail server like hMail , or even use Windows 2003 POP3 service for simply send/receive emails.
Use Source Control
Never ever develop source codes in an uncontrolled environment. Even a single developer, should use a sort of soruce control program. For example, if I modify a .cs file and damage the code for some reason, with a source control program, like SubVersion, I can rollback the changes. For small to medium projects or offices, I recommend to use SubVersion, which is a free source control program that supprots many important features like versioning and branching. If you have several product lines, projects and you have money! , I highly recommend you to install and use Team Foundation Server. TFS has great features for pushing a software development process, including a source control system. The default TFS client is Visual Studio (Team Explored) but there are other 3rd party clients so that if you are not using .NET , you don’t have to forget TFS.
Apply a standard software development process
There are several software development processes that have their pros and cons. Overally the value of each one is revealed due to the context of your projects, the size of your company, your budget and etc. If the size of the company, the size and number of projects and the number of people involved are too many, you are highly recommended to choose a SW development process that complies with CMMI, like RUP and MSF. CMMI is a kind of standard/framework for software development process and has five levels. If you manage to install and supervise a CMMI-matching process completely with constant improvements, you are likely to be a CMMI Level 3 !
I personally prefer MSF (Microsoft Solution Foundation) since despite RUP, which is extremely comprehensive and suites large organizations, MSF has different versions for Agile software development and CMMI improvement that takes you from CMMI Level 3 to Level 5 seamlessly. Also, Team Foundation Server works based on MSF-based processes so that you won’t need to think about a software to make the process running.
Regardless of the process instance you chose for your company, you should be aware that you take the following subjects into account:
Business Modelling: Begins from the first time a customer contacts you. The act of receiving the needs and requirements, analyse them and giving feedback in form of a Vision document is called Business Modelling.
Requirement Management : It means how you get your customer’s and business analysts’ needs, discuss and filter them and send them to development team. This stage can be mixed with Business Modelling under circumstances in which time and budget are too tough.
Analysis and Design: This stage is my favorite and (in my opinion) the most important part of a process. In this level requirements are received, analysed into details. Based on the output of the analysis software design is performed. In agile methods it is very common to omit the design part or blend the two. At a minimum level Use Case (or whatever they are called in your process) of important / major requirements must be modeled.
Implementation: this step includes coding, unit testing etc. Before getting involved in implementation all your standars about checkin/check outs, change management, coding & testing & other standards etc must be cleared.
Configuration & Change Management : How your source codes are maintained, how you grant or deny access to users, how you organize different branches of source code and …. For example if you release first version of a product, you label and call it “1”. Then until you release the 2nd version, you create a branch called “2” for service packs. New features are added to “1” but urgent changes and bug fixes are performed on “2”. When you are ready to release the 2nd version you merge “1” and “2”. Change management is responsible of receiving the change requests from the customer and manage them. Usually a team called CCB decides about which requirement should be implemented and when.
Project Management : How you assign tasks, plan your project, keep the track of time, manage closings etc.
Testing : What tests are performed, when and how?
Deployment : Also called Release Management. Tells us how to prepare deployment environment, how to build deployment packages and etc.
If you are not going to have a software factory and there will be only a few projects to work on, you are invited to use more lightweight agile methods such as XP and SCRUM. Unfortunately in many cases being agile is intepreted as having no process and no standards. This ends to a product with no proper testing, no documentation and no versioning.
Always put your customized document templates somewhere accessible to team members with sufficient privileges.
Launching a development team requires some concidetations, no matter your project or company is too small or too large. In this post I poited to some of them and told you how to do something urgent about them.