Linux

Understanding Cache-Control Headers

What is a header?

Whether you’re using a caching mechanism on your site’s server or not, Cache-Control headers are important to your site’s scalability and your end-user’s experience. Caching, or storing a copy of the completed request, can drastically help limit what requests your web server actually has to serve. But before we dive deeper into headers and cache, let’s first define: what are headers?

Headers are bits of contextual information your web browser can send and receive with requests for pages and files. Certain headers are sent with a request, while others are associated with the response you receive back. In this article we’ll be focusing on the “response headers” specifically. However, some of the directives we talk about are available for “request headers” as well.

With each request sent on the web, headers are returned. Here’s an example of the headers I get from my own domain:

The response headers tell me some information about my request.

HTTP/1.1 200 OK – This is the HTTP response code. It tells me whether my request was successful (200), redirected (301/302), received an error (500), or forbidden (403). There are tons of different status codes you might receive. Check out the full list for more information.

Server: Nginx – This section tells me the type of server that responded to my request. My site is hosted on a stack that uses Nginx as the web server software.

Date: Thu, 10 Aug 2017 17:00:29 – This is simply the date and time the response was served.

What are Cache-Control headers?

Sites may include many other kinds of headers, but I want to specifically call out two more:

Cache-Control: max-age=600, must-revalidate – This header defines how long my page should be cached for. The number is in seconds, so this example indicates a 10-minute cache time.

X-Cache: MISS – The X-Cache header indicates whether my request is served from cache on the server or not. MISS means my request is not already in cache. It passes to the web server to process, then stores a copy in cache on the way back. The next time I hit my site’s home page within the 10 minute cache window, it will serve it from cache.

What kind of cache are we controlling?

Cache-Control headers are responsible for controlling all caches, including (but not limited to): public, private, and server-level page caches. The Cache-Control header in my example above tells my web browser (Chrome in this case) how long to keep the response in its local browser cache.

It also tells the Varnish page cache on my server how long to cache the page.

The difference between the two is where the cache exists. In my Chrome browser, I’ve told Chrome to cache the page for 10 minutes with this response header. But what if I purge the cache in Chrome? Varnish page cache on my web server has still cached the response for 10 minutes. So I may still see the cached result from Varnish, unless I purge the page cache on the server too.

What kind of caching directives can I use?

There’s a long list of directives accepted for Cache-Control headers. Some are accepted only for Response headers, while others are also accepted on Request headers.

“public” or “private” directives are accepted only in Response headers. The “public” directive means the response can be cached by any service, regardless of whether you have HTTP Basic Authentication in use on the site or not. Meanwhile, “private” says to only cache in the browser and not in persistent caches, like my server’s Varnish page cache. It means the information on that page should only be cached for that user and not others.

“max-age” directives tell the caching mechanism how long to cache the response before it is considered “stale.” In my example the “max-age” is set to 600 seconds, or 10 minutes. The max-age directive can be used in Request and Response headers.

“must-revalidate” says that once the cached response is considered “stale” (after 10 minutes in my example), it has to be re-fetched/served as new. This directive is only accepted on Response headers.

“no-cache” and “no-store” relate to whether the cached response has to be re-validated by the server (check if the response is the same), and whether the response or content has to be re-downloaded. If “no-cache” is present, the cache mechanism cannot serve the cached response. Instead it must re-check if the response is the same. And if “no-store” is present, the cache mechanism cannot store the data/downloaded content and has to re-fetch the content again. These directives are accepted on both Request and Response headers.

Where do I set these headers?

There are several places you can define caching directives. If you’re one of many on the web using an Apache web server, you can set these directives in the .htaccess file. With an Nginx web server, you can set this in your Nginx configuration. If you use both, you’re probably using Nginx to handle all requests and pass certain ones to Apache. This means you should use the Nginx method of setting these headers.

If you don’t have access to the .htaccess file or Nginx configuration for your site, there are plugins out there which can configure this for you. W3 Total Cache or WP Super Cache both work great for this. If you’re using a host that offers server-level page cache like WP Engine, they take care of the configuration for you.

Apache or Nginx: The Web Server Showdown

Whether to use Apache or Nginx in your server stack is a decade-old question. While both web servers are versatile and open-source, both offer some pros and cons worth considering. Below I’ll break down some of the key differences between the two.

Architecture

One of the first aspects to consider is the way each web server handles requests. On Apache, there are several “multi-processing model” options to choose from. The most common is called “prefork” – in this model, there is a parent Apache process, with multiple “child” processes. With each new request that comes to the site, it opens a new “child” process. This allows your server to kill child processes in the form of individual requests, rather than having to kill the entire Apache service.

With Nginx, the master process exists with any number of “worker” processes. However, the largest difference comes in that each “worker” can handle multiple (read: thousands) of page/resource requests at a time, without having to engage the other worker processes. This frees up the server resources that these worker processes would have used, to allow other services to use the Memory and CPU more dynamically. Nginx’s architecture allows the server to process high amounts of traffic while still leaving Memory unused, compared to Apache’s single-request-per-thread method.

Compatibility

When Apache was built in 1995, it was an early success. Developers chose this web server model because of its flexibility and wide library of dynamically-loaded modules to extend its capabilities. By a year in, Apache dominated the web server markets by far. Because of its wide adoption rate, Apache documentation is vast. Many other software models integrate with and support Apache as a result.

Igor Sysoev originally developed Nginx in 2004 to be a web server to be used in combination with Apache. It was launched as a solution for sites which needed to serve thousands of simultaneous requests, which was a new horizon for the world wide web. As Apache’s adoption grew steadily, more modules extending its capabilities were released. When developers realized how lightweight the architecture ran on their hardware, it was an easy choice to begin using Nginx for static and dynamic requests.

As of today, Apache still holds about 50% market share of web servers, while Nginx holds about 35%.

Performance

When considering performance for Apache or Nginx, two main categories exist: static files, and dynamic content. Static files are files which don’t have to be regenerated when requested: images, css, fonts, and javascript. When your code constructs a page to be served, the content is dynamic. Other factors to consider here are: concurrent users, Memory used, transfer rate, and wait time.

In a base comparison of static files, Nginx is about twice as fast, and uses about 25% less memory:

And the same tester found that with dynamic requests, the web servers returned the responses in the exact same amount of time, when serving 16 concurrent requests:

However, remember the other aspects: what was the transfer rate? Do these results change with the number of concurrent users? Below is a comparison published by The Organic Agency:

You can see as the concurrent dynamic requests increase, so does the Apache response time. Nginx by contrast does show a load time increase as the concurrent users increase, but not nearly as much as Apache. Also consider the “failed requests” column which begins at about 25 concurrent users with Apache. Remember the architecture differences in how the servers handle multiple requests? This test is a clear indication of why Nginx is the model of choice for high-traffic environments.

Why not both?

Having trouble deciding whether you should use Apache or Nginx? Need versatility AND agility? No problem. There’s no reason you can’t use both web servers in your server stack. One common way to use both together is to use Nginx as a front-end server, and Apache as a back-end worker. This setup works because the performance disparity in the services is less noticeable for dynamic requests.

Using both servers allows Nginx to act as a “traffic director,” handling all requests initially. If the request is static, Nginx simply serves the file as-is. If the request is dynamic, Nginx uses a proxy_pass to send the request to the Apache web server. Apache processes the PHP and queries to generate the webpage, and sends it back up to Nginx to serve to your site’s visitor. If you’re looking to set this up on your own, check out Digital Ocean’s documentation on the subject.

4 Important Ways to Identify a Botnet

What is a botnet?

A botnet is a series of internet devices (computers, websites, servers, modems, mobile phones, and more) infected with malware, harnessed together. Their purpose is to infect more devices with malware to build their network even further.

Botnets are sometimes very hard to identify because of their nature to switch IP addresses frequently, so as to not trigger security software to take action. Furthermore, botnets can “spoof” legitimate browsers in their requests, making it seem like the request is coming from a normal web browser or mobile phone.

These attributes aside, there are still a few key tricks you can use to identify a botnet.

Bots don’t load assets

Try visiting your site, then look through your site’s access logs. Locate the requests associated with your IP address. Do you see a “GET /” entry in the log? You should! But you should also see a number of static files requested after that, like requests for your theme’s CSS and JavaScript files.

One key attribute of botnets is that they do not load these CSS and JavaScript files. For these you will ONLY see a “GET /” entry, but no static files requested. What’s tricky about this is because botnets don’t load Javascript, these visits to your site are invisible on your Google Analytics reports. You’ll need access to logs from your site’s server itself to identify a botnet.

Bots make illegitimate POST requests

Another way to identify a botnet is to look at the behavior of the IP address: they will often try to make “POST” requests instead of just “GET” requests. A POST request is a request intended to update information, or login to a site. In WordPress, the most common places for this would be your /wp-login.php page, and your /xmlrpc.php page.

The /wp-login.php page is where you and your site authors will log into your WordPress site. A POST request here comes from someone trying to log into the site. On WP Engine, POST requests to the login page without loading the page’s assets are blocked, meaning you are safe from being infected by malware. But, these botnets may be bothersome all the same for various reasons.

Similarly, the /xmlrpc.php page is another page targeted by botnets. This page only accepts POST requests from mobile posting apps, so that you can remotely update your site. Botnets will target these pages in hopes of some kind of security exploit, so they can infect your site and server with malware. On WP Engine, requests to XMLRPC are only accepted from legitimate users on your site, which is another way they protect you from this kind of traffic.

Bots may spoof “fake” browser versions

In your site’s access logs, you’ll generally see entries like this:

172.56.7.237 techgirlkb.guru - [07/Aug/2017:02:32:11 +0000] "GET /2017/08/quick-wins-database-optimization/ HTTP/1.1" 200 9064 "http://m.facebook.com/" "Mozilla/5.0 (Linux; Android 7.1.2; Pixel XL Build/NKG47M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/59.0.3071.125 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/135.0.0.22.90;]; WPEPF/1"

This section in quotes “Mozilla/5.0 (Linux; Android 7.1.2; Pixel XL Build/NKG47M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/59.0.3071.125 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/135.0.0.22.90;]; WPEPF/1” is the “user agent string” for the request that came in. Normally with a long string like this, it looks like a pretty legitimate user. But often times, botnets will use a user agent string that looks legitimate, but is just slightly off. Here’s an example of one I see a lot:

"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1"

This one is really tricky. Based on the results of my user agent string lookup, it appears to be a Windows 7 user, using Firefox version 40.1 as a web browser. Only, Firefox doesn’t have a version 40.1. You can find the real versions of Firefox on their website. This is another way in which it can be hard to identify a botnet: they evade normal security software and firewalls by pretending to be legitimate user agents.

Botnets will change IPs after an invalid response

Last, the other key behavioral trait of a botnet is that it will rotate IP addresses. Usually this happens when the botnet detects it’s been blocked. For instance, if it receives an empty response or a 403 Forbidden response, this signals the botnet that it’s time to move on to a different IP address from another one of its malware-infected devices in the network.

There are several known Internet Service Providers and cell carriers whose devices are most vulnerable to this kind of botnet and to malware. You can read more about it in this comprehensive study done by WordFence. You’ll notice if you do an IP lookup for the IP addresses making these requests, that most are from obscure countries, with the same telecom provider.

One way you can battle the rotating IP addresses is by limiting the traffic allowed to your site to only countries you know to be legitimate. Or (if that’s going to be a really long list), you can alternatively block traffic from the countries you see these botnet requests originating from.

What are IOPS and Why are They Important?

In every engineer’s life, there comes a time when your company decides it’s time to choose a new web host. This process can go smoothly, or you could crash and burn. The difference in the outcomes often comes down to: IOPS.

What are IOPS?

So what are IOPS? IOPS stands for Input/Output operations per second. This is an allowed value pre-determined by your webhost, which controls how many Input/Output operations are allowed at once on your server. After the threshold is reached, the host may begin throttling these operations, causing requests and processes to queue up. This in turn causes sleep-state processes, which inflates your server load more and more, until the backlog queue of requests finally completes. The processes left waiting in this time are affected by “IOWait.”

So why are IOPS important?

With that in mind, it’s first important to understand how many IOPS your site really needs. Start by getting some stats from your current environment. How long does it take your current server to process requests? How many CPU cores and how much Memory is allocated on your current environment? At your daily peak for CPU load, how many active processes were running and how fast were they? What is the ratio of disk reads to disk writes?

What IOPS benchmarks are important?

Second, don’t take the IOPS metrics different companies boast about too seriously. IOPS can vary drastically depending on the type of operation, size of the server block, number of requests, and weight of those requests. It would be easy for a vendor to falsely inflate the IOPS their systems can handle by showing a dedicated block with lightweight reads and writes. It’s important to ask about how the benchmarks were made. What was the read/write ratio? Were there other tenants on the disk? Is there a set point at which requests are throttled?

Getting specific

It might be more appropriate to develop a test instance of one of your heavier-trafficked sites on a new web host, and perform load testing to see how the type of traffic your site normally experiences performs on the new environment. The more specific to your own needs the testing can be, the better. Be sure to check with the vendor on their policies for load testing before getting started.

Linux and Page Faults

What are Page Faults?

If you’ve ever been monitoring a Linux process you might notice an odd metric: Page Faults. On the surface Page faults can sound scary. But in reality, a page fault is the natural process of requesting physical Memory to be allocated to different processes.

Linux divides up its physical Memory into “pages,” which are then mapped to virtual memory for individual processes. Each “page” equates to about 4kb of physical memory. And, just because the pages are mapped to a certain process, this doesn’t necessarily mean the process is actively using those pages. There can be both inactive and active pages in use by a process on the server.

So what is a page fault? Don’t worry, it doesn’t really have anything to do with errors. A page fault simply happens when a program is executing some code, and the executable code for that program isn’t currently being actively held in physical pages of Memory. Linux responds by allocating more pages to that program so that it can complete its execution of the code.

Minor Page Faults vs Major Page Faults

You may also see a shockingly high number of “minor page faults” in monitoring your process. Minor page faults are even less worrisome than major ones. This simply means a process requested data that’s already stored in Memory. The Memory just wasn’t allocated to the process requesting it. Linux will remap the currently used data to be shared between the two processes, without having to access the physical memory. The easiest way to remember the difference is: minor page faults can be served from existing virtual memory, while major page faults have to request more pages be allocated from the physical memory.

Page Faults and Swap

So what happens if your server has already mapped all its pages of the physical memory to different processes, but more Memory is needed to perform a task? The system will begin “writing to swap.” This is not a great scenario in any situation. The system will write some of the pages it’s holding onto in Memory to the disk. This frees some pages to satisfy incoming page fault requests.

Compared to using virtual Memory, writing pages to disk is super slow. And, writing to disk is an IO operation which can be throttled on many hosts if it happens too much. In regards to page faults, the process of using swap is the most worrisome. Using swap can easily become a downward spiral! Pages requested are written to disk, which overwrites other pages requested previously. But when the previous pages are requested again the cycle repeats itself again and again, until your system crashes.

With that in mind, the important aspects of your system to monitor are not necessarily the major or minor page faults, but rather the amount of swap being used.

Have you experienced issues with swap usage or page faults on your own system? Tell me about it in the comments.