June 2020 Newsletter
Everything Old Is New Again
Search and Website Redesign
If you are planning a website redesign, whether it is an upgrade or changing content management systems, we encourage you to think about its impact on your search results. A good way to start factoring search into your redesign project is by letting us know ahead of time of your plans and if possible, the timeline you have set for them. Being aware of your redesign plans allows us to work together with you and prepare for this transition.
To get started, we suggest you read our checklist for a successful website redesign.
Don’t Get Locked Out
Take Control of Your Search.gov User Account
Now our help manual includes step-by-step instructions on how to create and edit your user account with the Login.gov service.
SEO Pro Tip
Robots.txt Crawl-Delays Aren’t What They Used to Be
Robots.txt files have traditionally included an option to direct spiders to crawl your website at a pace of your choosing, the crawl-delay, though it’s not part of the formal protocol. A robots.txt file for a Drupal website defaults to a 10 second crawl-delay. However, Google announced last year that they would stop honoring crawl-delay, due to widespread misconfiguration issues.
We do honor crawl-delays, by popular demand, but as we point out on our Robots.txt explainer page, they have a significant impact on the length of time it takes to index your content, especially for large websites. We plan to keep honoring crawl-delay settings for the foreseeable future, but ask that you add a provision that would allow us to work more swiftly in your site. We will never work faster than one request per second, but if you want a slower pace we suggest a two second crawl-delay. For example: user-agent: usasearch crawl-delay: 2.
Release Notes
Want to learn about the latest features, fixes, and focuses of the Search team? We post monthly Release Notes on our website. Here are some highlights:
- We are thrilled to report that all the major system upgrades we’ve been working so hard on this year are complete!
- We’ve upgraded all of Search.gov’s elasticsearch components and are now operating on Elasticsearch 6.8.
- We now have routed queries available for our results API users. If you use the API and have questions about how to implement this feature on your site, reach out.