June 2021 Newsletter
Summary
Please look for an email that contains our annual survey. We have good news to report about Akamai. The indexing updates process explained.
Annual Survey
Your Feedback Requested!
Search.gov is always seeking feedback to help us serve our customers better - help us help you!
Please look out for a survey in your inbox with the subject like “GSA Customer Survey” - it will have a link to our annual survey. We use these results to better understand how the program is doing, and how we can better meet your needs. The survey is only open until July 9, 2021, so please respond today!
Akamai Update
Allow-listing no longer required
As many of you will remember, our servers have faced challenges working through Akamai’s firewall and bot detection services over the years. To accommodate, agency teams had to request we be added to their allow-lists, otherwise we would be flagged by Akamai and blocked. We’re pleased to report that after working closely with Akamai, our servers will be able to work freely inside any website using their services, without teams having to modify their allow-lists.
🎉
Indexing Content
New Pages, Updates and Removals
We visit sitemaps every two hours, picking up urls that weren’t in our system before. At the same time, we scan the sitemaps for any updated <lastmod> timestamps on urls we already knew about, and refetch those to get the updates.
Separately, we have a process that will recheck each URL once it’s been 30 days since we last fetched it. By doing this, we’re able to pick up updates to any pages that didn’t get a new <lastmod> date on a sitemap, and find updated response codes, like 301s or 404s, and remove those urls from the index. Usually, URLs that are 301ing and 404ing are not included on sitemaps, so we have to detect them separately.
Important note: Removing a URL from a sitemap will not automatically remove it from the index if the URL still returns a 200 OK response. If you want to remove a live page from our index, you’ll need to place a robots noindex tag on it, either in the <head> or in the header response. If you want a live page indexed by commercial search engines but not by us, please email us to have it manually removed.
Service Update
Results Page Redesign
We’ve made great progress with the initial phase of development for the Search Results Pages (SERP) redesign - removing legacy code. The removal of this legacy code will help clear the path for future redesign development. Thanks to all of those who have helped with usability testing for our refreshed SERP design so far - we’ll continue to provide updates as we make further progress!
Release Notes
Want to learn about the latest features, fixes, and focuses of the Search team? We post monthly Release Notes on our website.