TechGirlKB

Performance | Scalability | WordPress | Linux | Insights

Posts

How To: Ask Google to Recrawl Your Site

What does “crawl” mean?

Before we get started, we should probably clarify: what does it mean when Google crawls your site? Google crawls all pretty much all publicly-accessible websites. This is a search-engine’s process of finding your site’s links and URLs, and indexing them for others to find. Generally, search engines listen to rules you set in your “robots.txt” file. In WordPress, you can allow search engines or discourage them on your Settings > General settings screen. After you’ve launched your site, you’d generally want Google to crawl their site.

What if I don’t want Google to crawl my site?

The one exception is if your site is not public to all web users. It technically is possible for Google to crawl a non-publicly accessible site. This generally happens if a link to that site appears on another site which is publicly accessible. In this case, you can request the URL be removed. Remember, this is temporary. So you still need to update your preferences to tell Google not to crawl at all. In the 90 days between the URL removal and reindexing, you need to be sure to add a noindex header or meta-tag to ensure it’s not indexed going forward.

When would I want Google to recrawl my site?

There are potentially a lot of scenarios when you’d want Google to take another look at the URLs that exist on your site. Some common examples include:

  • You’ve recently rebuilt or restructured your site
  • You’ve added a large number of redirects on your site
  • You’re seeing a lot of 404s reported in Google’s Webmasters Tools
  • You’ve been hacked or compromised and some bad URLs got indexed
  • Google crawls or indexes links that don’t exist

In these cases, it would be appropriate and advisable to ask Google to recrawl.

How do I ask Google to recrawl my site?

First, I’d recommend updating your sitemap is current, and submitting that to Google. Remember: if you use Yoast SEO, the plugin updates your sitemap as you go. Once you’ve confirmed your sitemap is up to date, be sure to add it to Google (and remove any old ones).

Be sure to click any old sitemaps in the list, and remove them before continuing.

To ask Google to recrawl from here, you can follow these steps:

  1. Go to Google Webmasters Tools and sign in.
  2. Select the site you’d like to recrawl from the list.
  3. Click “Crawl” from the menu on the left.
  4. Click “Fetch as Google.”
  5. Leave the box empty and submit. 
  6. Wait a few seconds as Google fetches and renders your site.
  7. When it finishes fetching, click “Request indexing.”
  8. In the popup window, click “Crawl this URL and its direct links.”
  9. Wait for a few days – Google usually takes several days to recrawl URLs submitted.

After these steps, the search engine results for your site will be more accurate! However, you’ll want to be sure you allow Google several days to do the recrawl.

 

That’s all, folks! If you’ve followed the above steps, you’ve successfully asked Google to recrawl your site. Future Google crawls should be more successful too, with your updated sitemap. Additionally, you can find instructions for other search engines online.

Have any other thoughts or advice on how to get Google to recrawl your site? Leave me a note in the comments, or contact me. 

 

How to Choose the Best CDN for Your Site

What is a CDN?

Before we breach the topic of choosing the best CDN for your site, first we should consider: what is a CDN? The term “CDN” stands for “Content Delivery Network.” It’s a network of servers located all over the world. When used on your site, CDN helps distribute your content across these servers. This means users around the world will receive your content faster!

Full Site vs. Static Assets

There are two main ways CDN is used on websites: full-site CDN, or serving only static assets. The static assets are your site’s images, CSS files, fonts, and javascript. Since most CDN providers charge based on bandwidth, generally full-site CDN will be more expensive. If you choose to use a full-site CDN, the layers of CDN cache will serve as a web server in front of your origin server. And this means your origin server would be used for only uncached requests on your website. While this would help offload a large amount of traffic from your origin server, it also means your site is hosted across many servers worldwide. So if one CDN server goes down, the rest of the world could still see your site.

If full-site CDN isn’t an option or isn’t ideal for your site, you can still use CDN to host the images and other static content of your site. This option still helps your site load faster for users worldwide, and will likely be less expensive than using full-site CDN. With CDN serving static assets, you should still use a proxy cache like Varnish on your origin server. This will help your site’s performance and lighten the load on your server.

choosing the best cdn

CDN Comparison

There are a number of studies released comparing different CDNs, but in my mind there are a few key metrics to focus on: pricing, cacheability, number of web nodes, SSL support, and whether they offer full-site vs. only static resources. Both MaxCDN and Stratusly published similar reports on these factors in 2016. Below are the comparisons from Stratusly, who added more factors to consider:

Who supports full-site CDN?

Akamai, Fastly, Amazon CloudFront, and Verizon (formerly EdgeCast) offer static and dynamic caching options. These setups typically require pointing the nameservers from your DNS provider to them. This is known as a reverse-proxy. Fastly in particular supports “event-driven content” which they consider to be neither dynamic or static. This allows them to cache more effectively.

Cloudflare and MaxCDN (now StackPath) do not offer full-site CDN.

Who has the most web nodes worldwide?

Akamai is the clear leader here by far. They offer 2700 web nodes worldwide, while the closest competitor (CloudFlare) has 76. Akamai also isn’t able to provide a specific range of IPs if requested. The IPs in use are constantly changing. This makes it incredibly difficult for any one source to target them with a DDOS.

Who is the least expensive?

This is a more difficult question to answer directly. Some pricing models vary based on the amount of traffic, and some are flat-rate. MaxCDN made this visual representation of how Amazon’s pricing model compares to theirs:

And, here’s the graphic the Stratusly team made to compare the three whose pricing varies on bandwidth:

The pricing tends to look pretty similar in the beginning, until you look at enterprise volumes of traffic. If you look at high volumes of traffic then MaxCDN looks great, but also consider that they do not offer full-site CDN.

Who has the best support?

Of the six competitors we’re comparing, only two don’t include Free 24/7 Support. CloudFlare offers phone, chat, and email support 24/7 for Enterprise-level accounts, but only email support for the rest of their plans. For Amazon, they offer a wide library of online support guides and forums, but live support is not available unless you choose one of their Business or Premium Support plans.

Who supports SSL?

All the competitors offer “Shared SSL,” which is a way of saying they will secure the CDN URL they provide you. For example, on MaxCDN they use subdomains of the same root: somezone.company.netdna-cdn.com. You’re able to enable the “Shared SSL” to secure this URL. However, many companies will want to use a Dedicated SSL (SNI) based on a Custom CDN URL of their choosing. Amazon, CloudFlare, and MaxCDN offers these options. None of these, however, support HTTP/2 (a faster version of HTTP). MaxCDN is closer since they offer SPDY support.

Who is the fastest?

All the competitors offer differing metrics here, claiming to be the fastest. These metrics may mean nothing to you though, depending on your options, your geographic location, and the location of your users. I’d recommend using CDNPerf to compare usability and speed worldwide. This comparison spans 24 CDN providers and compares worldwide, specific regions, or specific countries to help you make the right decision. You can also use their “compare CDN” tool to view performance graphs.

Conclusion

So which is the best CDN for you? It highly depends on your company’s needs for CDN. Be sure to consider all the factors before making your decision. Compare performance, pricing, features, and support before deciding.

Are there other features worth considering when picking a CDN provider? Do you have other thoughts about CDN? Feel free to let me know in the comments or by contacting me.

 

Brute Force Attacks and WordPress

Introduction

WordPress is a leading Content Management System used by around 27% of all websites. As its market share grows, more and more attacks target WordPress sites. A common type of attack is called “Brute Force.” In this method, the attacker simply guesses combinations of usernames and passwords repeatedly until they find success. Most often, these attacks are made by bots.

There are some simple ways to avoid penetration by a Brute Force attacker. Prevention is managed at two primary levels: Firewall-level, and WordPress-level.

brute force attacks

Firewall-level prevention

One of the best ways to prevent Brute Force attacks is using a firewall. If you manage multiple sites on the same server, the protection you put in place at this level will protect all your sites. Additionally, if attacks like this can be mitigated at the server-level, this protection won’t have to be processed by PHP/WordPress, which means fewer server resources are being used.

Cloudflare

CloudFlare is a reverse-proxy service. In this scenario you direct your site’s nameservers to point to CloudFlare, and manage the A records or CNAME records from your DNS provider at the CloudFlare level. This option is ideal for a few reasons. First, CloudFlare masks your site’s “Origin IP,” or the IP address of the server where your site’s content is hosted. The IP masking prevents attackers from sending attacks directly to your server and bringing your site down.

On top of the natural benefits of using this service, CloudFlare offers a rate-limiting option. Using “Protect Your Login,” you can configure rules that will block IPs if they have made POST requests to your login page within the last 5 minutes.

The benefit of blocking the IP addresses with CloudFlare is that the attackers don’t even have to touch your site to get denied. So, no server resources on the server where your site is hosted will be used.

Sucuri WAF

Sucuri WAF works similarly to CloudFlare in that it offers protection as a reverse-proxy. Using Sucuri WAF will also mask your site’s origin IP, and block IP addresses before they can reach your origin server. The difference between the two is that Sucuri is a leader in website security. Meaning, their service is geared specifically towards security, protection, and compatibility. Additionally, they offer more configuration options specific to security overall.

HiveWind

HiveWind is a DDOS Mitigation service that sits in front of your site’s server as well, as a Cloud Load Balancer. Their services cover a large number of attacks that might affect your site, and the layer is cloud-based. The HiveWind firewall can automatically detect bad-actors like Brute Force attacks and botnets. The large difference between HiveWind and its competitors is that it cumulatively blocks attacks. That means if an IP has attacked another site, it’s blocked for all users on HiveWind’s service. And unlike many other DDOS services, even the enterprise-level services are flat-rate no matter the scale of the attack.

If you want to try HiveShield DDoS protection on your own server, use the coupon code TCHGRLKB. This coupon code is good for 8 cores/$50 a month OR 16 cores/$100 a month, each with a free 30 day trial – a 50% savings!

Incapsula

Incapsula is a reverse proxy system used as a CDN to sit in front of your origin server. While the setup process can be trying sometimes, the end result is a thoroughly-secured website. Their IncapRules security system uses advanced detection to identify whether the user is a bot, and block sessions. And, what’s unique about Incapsula’s system is that it allows you to configure the protection to be as aggressive as you want. They also offer a resiliency score to test whether your site is ready to handle a DDOS or not.

WordPress-level prevention

If firewall-level protection isn’t possible, you can begin looking at WordPress-level protection. While this kind of protection leaves your server resources more vulnerable, it’s still helpful. The attackers will still be denied. But, making WordPress do the heavy-lifting is more taxing on your server.

Below we’ll cover some common options for your WordPress site. These options are still helpful to block potential brute force attackers on your site.

WordFence

The WordFence security plugin is an easy way to automatically block attackers. Using their settings, you can force users to set strong passwords, lock users out after failed attempts, and automatically ban users who try common usernames. One of the most common usernames is simply “admin.” With this plugin you can block anyone who tries this user. Read their blog post for more information.

Sucuri Security

Using the Sucuri Security plugin you can block Brute Force attacks. And not only this, you can also use their services to scan for malware and actively track file changes. What’s unique about this plugin is the level of logging it offers. Plus, the logging doesn’t go to your local database. It’s stored securely with Sucuri themselves. The Brute Force detection is best when used with their WAF mentioned above.

iThemes Security

The iThemes Security plugin (previously known as Better WP Security) is one of the widest-used security plugins. It offers 30 different ways to secure your site. One of these options includes moving your login page to a different URL. Since many Brute Force attacks rely on the login page being called “wp-login.php,” this alone can defer many attacks. Like HiveWind, iThemes security uses a brute force detection network. So, if one IP is blocked on another site, it’s blocked for yours too.

All-in-One WP Security

The All-in-One WP Security plugin offers a cookie-based detection of bots and brute force attacks. Since bots generally do not load assets like cookies, javascript, or css, this allows them to block bad actors. And, unlike other plugins, this one allows you to block attackers by IP address or user-agent. It also includes a “captcha” which makes the user prove they are human.

Hide Login Plugins

There are a number of plugins which will rewrite your login URL. Some common examples include: WPS Hide Login, Rename WP Login, and Loginizer. A single-function plugin like this is not always the most ideal. For instance, if you use page caching on your site, you’ll need to ensure your new login page is uncached. But, these plugins will deflect some brute force attacks simply because bots won’t know where to login.

WordPress Best Practices

Last, to prevent Brute Force attacks you should follow some simple best practices. This list will help prevent many kinds of attacks.

Don’t use the username “admin.”

This is the most commonly-used username in existence. Since Brute Force attackers need to guess your username and password, using this username gives them half the equation right away. Be smart! Choose a more unique username.

Use a captcha.

Captchas are usually image or math-based forms for testing whether a user is a human or not. Since most brute force attacks come from bots, this simple trick will prevent most attackers.

Use 2-factor authentication.

This method means a bot would have to guess two sets of authentication, one of which is constantly changing. Google Authenticator is one of the most common options. Be sure to install the app on your phone as well! This is how the system works for authentication.

Require strong passwords.

For any high-level user on your site (like an Author or Administrator) you should require they use a complex password. These days, adding a number to the end of a word won’t cut it. You’ll want to make sure your password is long, and includes many combinations of numbers, letters, and characters. Use these recommendations when choosing a password.

Keep everything updated.

I can’t stress this point enough. The most common source of malware on sites is outdated software. New exploits in software are being found every day. Most plugins and themes will release a patch or update as soon as one is uncovered. So this means keeping everything updated is super important. This certainly includes WordPress itself. If you manage many sites, using a system like MainWP or ManageWP can help you manage updates from a single dashboard.

Never log in from a public computer.

Public computers at your library or internet cafes are not the most secure. Casual hackers may have installed software on these computers that records everything you type. You should also never choose for a public computer to remember your password.

Use SSL on your login page.

Last, using SSL encryption on any page where you enter a password is important. This encrypts the data you send over the network between your local computer and your website. Any bad actors listening on your network won’t be able to read encrypted information.

Conclusion

So, there you have it! A comprehensive list of methods to protect your WordPress site. And, with these quick and easy methods you can effectively prevent Brute Force attackers from accessing your site. Remember also that using a firewall or plugin isn’t everything. You also need to check the list of best practices to secure your site. With all these powers combined, you can be sure your site is safe.

Do you have more tips and tricks? Have more thoughts on Brute Force attacks? Comment below or contact me to talk more about security.

How To: Configure Varnish for Responsive Sites

Scalability vs Responsive Sites

A major part of performance with sites on any platform is ensuring that your website is highly cacheable. By caching pages to serve as static files to repeat users, this offloads a lot of server processing power. Without cache, your web server would have to re-process the code to generate your page. Not only does this make more work for your server, it could slow down the response time for your site.

A challenge developers face is how to cache their sites heavily, while still adapting to mobile devices. With Google PageSpeed’s score hinging on mobile compatibility, many themes in WordPress are adapting to offer responsive options. Chances are you’re wanting a different, mobile-friendly version of your site to show to your smartphone users. With page cache, your user sees whatever version was requested the last time the page was stored in cache. This equates to a poor experience for your users.

X-Device Cache Groups

So how do you handle this scenario? Difference circumstances require two different versions of the site. You can do this by adding an “X-Device Group” header based on the HTTP User Agent the request came from. This allows you to detect devices in Varnish. This way, any device you designate as “mobile” gets cached in a mobile group. At the same time, desktop users are cached in a separate cache group. So this solves our problem! A mobile user receives the mobile site. A desktop user receives the desktop site.

Example Config

So how do you detect devices in Varnish? Here’s an example setup:

 

# Cache different devices in different cache groups so they're served the right version
if ( req.http.User-Agent ~ "(?i)(tablet|ipad|playbook|silk)|(android(?!.*mobile))" ) {
# Cache tablet devices in the same group - it's possible this is different than smartphones for some sites
set req.http.X-Device = "tablet";
}
elsif ( req.http.User-Agent ~ "(?i)
Mobile|iP(hone|od)|Android|BlackBerry|IEMobile|Kindle|NetFront|Silk-Accelerated|(hpw|web)OS|Fennec|Minimo|Opera M(obi|ini)|Blazer|Dolfin|Dolphin|Skyfire|Zune" ) {
# Cache smartphones together. The mobile user-agents receive a mobile version with a responsive theme
set req.http.X-Device = "smartphone";
}
else {
# Cache everything else together. We assume anything that doesn't fit into the above regex is a regular desktop user.
set req.http.X-Device = "regular";
}

 

This snippet says to add a X-Device header to each of these device groups. It allows Varnish to cache the devices separately. You can set the header to whatever you prefer. Some prefer something more like X-Cache-Bucket, or X-Cache-Group (which WP Engine uses). If you’re curious how I came to this setup and these particular user-agents, I grabbed these from a pretty thoughtful list. And, Varnish published a very helpful guide on device detection that I snagged to get started.

When you detect devices in Varnish caching groups like this, it allows you to take full advantage of page caching, while still maintaining responsive behavior. That’s a win you can take to the bank! Using more cached pages equates to faster website load times.

Understanding Cache-Control Headers

What is a header?

Whether you’re using a caching mechanism on your site’s server or not, Cache-Control headers are important to your site’s scalability and your end-user’s experience. Caching, or storing a copy of the completed request, can drastically help limit what requests your web server actually has to serve. But before we dive deeper into headers and cache, let’s first define: what are headers?

Headers are bits of contextual information your web browser can send and receive with requests for pages and files. Certain headers are sent with a request, while others are associated with the response you receive back. In this article we’ll be focusing on the “response headers” specifically. However, some of the directives we talk about are available for “request headers” as well.

With each request sent on the web, headers are returned. Here’s an example of the headers I get from my own domain:

The response headers tell me some information about my request.

HTTP/1.1 200 OK – This is the HTTP response code. It tells me whether my request was successful (200), redirected (301/302), received an error (500), or forbidden (403). There are tons of different status codes you might receive. Check out the full list for more information.

Server: Nginx – This section tells me the type of server that responded to my request. My site is hosted on a stack that uses Nginx as the web server software.

Date: Thu, 10 Aug 2017 17:00:29 – This is simply the date and time the response was served.

 

What are Cache-Control headers?

Sites may include many other kinds of headers, but I want to specifically call out two more:

Cache-Control: max-age=600, must-revalidate – This header defines how long my page should be cached for. The number is in seconds, so this example indicates a 10-minute cache time.

X-Cache: MISS – The X-Cache header indicates whether my request is served from cache on the server or not. MISS means my request is not already in cache. It passes to the web server to process, then stores a copy in cache on the way back. The next time I hit my site’s home page within the 10 minute cache window, it will serve it from cache.

 

What kind of cache are we controlling?

Cache-Control headers are responsible for controlling all caches, including (but not limited to): public, private, and server-level page caches. The Cache-Control header in my example above tells my web browser (Chrome in this case) how long to keep the response in its local browser cache.

Source: developers.google.com

It also tells the Varnish page cache on my server how long to cache the page.

Souce: book.varnish-software.com

The difference between the two is where the cache exists. In my Chrome browser, I’ve told Chrome to cache the page for 10 minutes with this response header. But what if I purge the cache in Chrome? Varnish page cache on my web server has still cached the response for 10 minutes. So I may still see the cached result from Varnish, unless I purge the page cache on the server too.

 

What kind of caching directives can I use?

There’s a long list of directives accepted for Cache-Control headers. Some are accepted only for Response headers, while others are also accepted on Request headers.

“public” or “private” directives are accepted only in Response headers. The “public” directive means the response can be cached by any service, regardless of whether you have HTTP Basic Authentication in use on the site or not. Meanwhile, “private” says to only cache in the browser and not in persistent caches, like my server’s Varnish page cache. It means the information on that page should only be cached for that user and not others.

“max-age” directives tell the caching mechanism how long to cache the response before it is considered “stale.” In my example the “max-age” is set to 600 seconds, or 10 minutes. The max-age directive can be used in Request and Response headers.

“must-revalidate” says that once the cached response is considered “stale” (after 10 minutes in my example), it has to be re-fetched/served as new. This directive is only accepted on Response headers.

“no-cache” and “no-store” relate to whether the cached response has to be re-validated by the server (check if the response is the same), and whether the response or content has to be re-downloaded. If “no-cache” is present, the cache mechanism cannot serve the cached response. Instead it must re-check if the response is the same. And if “no-store” is present, the cache mechanism cannot store the data/downloaded content and has to re-fetch the content again. These directives are accepted on both Request and Response headers.

Where do I set these headers?

There are several places you can define caching directives. If you’re one of many on the web using an Apache web server, you can set these directives in the .htaccess file. With an Nginx web server, you can set this in your Nginx configuration. If you use both, you’re probably using Nginx to handle all requests and pass certain ones to Apache. This means you should use the Nginx method of setting these headers.

If you don’t have access to the .htaccess file or Nginx configuration for your site, there are plugins out there which can configure this for you. W3 Total Cache or WP Super Cache both work great for this. If you’re using a host that offers server-level page cache like WP Engine, they take care of the configuration for you.

 

Apache or Nginx: The Web Server Showdown

Whether to use Apache or Nginx in your server stack is a decade-old question. While both web servers are versatile and open-source, both offer some pros and cons worth considering. Below I’ll break down some of the key differences between the two.

Architecture

One of the first aspects to consider is the way each web server handles requests. On Apache, there are several “multi-processing model” options to choose from. The most common is called “prefork” – in this model, there is a parent Apache process, with multiple “child” processes. With each new request that comes to the site, it opens a new “child” process. This allows your server to kill child processes in the form of individual requests, rather than having to kill the entire Apache service.

source: zope.org

With Nginx, the master process exists with any number of “worker” processes. However, the largest difference comes in that each “worker” can handle multiple (read: thousands) of page/resource requests at a time, without having to engage the other worker processes. This frees up the server resources that these worker processes would have used, to allow other services to use the Memory and CPU more dynamically. Nginx’s architecture allows the server to process high amounts of traffic while still leaving Memory unused, compared to Apache’s single-request-per-thread method.

Source: www.thegeekstuff.com

Compatibility

When Apache was built in 1995, it was an early success. Developers chose this web server model because of its flexibility and wide library of dynamically-loaded modules to extend its capabilities. By a year in, Apache dominated the web server markets by far. Because of its wide adoption rate, Apache documentation is vast. Many other software models integrate with and support Apache as a result.

Igor Sysoev originally developed Nginx in 2004 to be a web server to be used in combination with Apache. It was launched as a solution for sites which needed to serve thousands of simultaneous requests, which was a new horizon for the world wide web. As Apache’s adoption grew steadily, more modules extending its capabilities were released. When developers realized how lightweight the architecture ran on their hardware, it was an easy choice to begin using Nginx for static and dynamic requests.

As of today, Apache still holds about 50% market share of web servers, while Nginx holds about 35%.

Performance

When considering performance for Apache or Nginx, two main categories exist: static files, and dynamic content. Static files are files which don’t have to be regenerated when requested: images, css, fonts, and javascript. When your code constructs a page to be served, the content is dynamic. Other factors to consider here are: concurrent users, Memory used, transfer rate, and wait time.

In a base comparison of static files, Nginx is about twice as fast, and uses about 25% less memory:

Source: www.speedemy.com
Source: www.speedemy.com

And the same tester found that with dynamic requests, the web servers returned the responses in the exact same amount of time, when serving 16 concurrent requests:

Source: www.speedemy.com

However, remember the other aspects: what was the transfer rate? Do these results change with the number of concurrent users? Below is a comparison published by The Organic Agency:

Source: theorganicagency.com

You can see as the concurrent dynamic requests increase, so does the Apache response time. Nginx by contrast does show a load time increase as the concurrent users increase, but not nearly as much as Apache. Also consider the “failed requests” column which begins at about 25 concurrent users with Apache. Remember the architecture differences in how the servers handle multiple requests? This test is a clear indication of why Nginx is the model of choice for high-traffic environments.

Why not both?

Having trouble deciding whether you should use Apache or Nginx? Need versatility AND agility? No problem. There’s no reason you can’t use both web servers in your server stack. One common way to use both together is to use Nginx as a front-end server, and Apache as a back-end worker. This setup works because the performance disparity in the services is less noticeable for dynamic requests.

Using both servers allows Nginx to act as a “traffic director,” handling all requests initially. If the request is static, Nginx simply serves the file as-is. If the request is dynamic, Nginx uses a proxy_pass to send the request to the Apache web server. Apache processes the PHP and queries to generate the webpage, and sends it back up to Nginx to serve to your site’s visitor. If you’re looking to set this up on your own, check out Digital Ocean’s documentation on the subject.