Since 2018, we've made an annual post detailing our infrastructure and planning efforts to serve your library, your staff, and your patrons after a natural disaster, and share the additional improvements we've made in the last year to our Disaster Preparedness and Recovery plans. We've spent a significant amount of time building infrastructure, reviewing procedures, and planning for an untimely incident. For this year's post, we starting with our most recent upgrades and working backwards to our original September 2018 update.

2021 Updates & Improvements

To prepare for this year's post, we asked Lee what he felt were important updates or changes made in 2021, or for any other info he wants the KLAS Users' Community to know about our efforts. The two items he mentioned are:

1. The recovery process is the the same as before, but note that restoration of databases is sequential (per server), not parallel. So, while a one-hour recovery for a database at top of the queue is possible others at the bottom of the list will have a longer wait as the recovery process works through the list.
2. We have added weekly server snapshots to our AWS backup servers. For an end user, this makes no difference to restoration, but it makes Keystone staff's jobs infinitely easier.

2020 Updates & Improvements

Much of what we put into place as part of our disaster preparedness plan in 2020 is what allowed our staff to begin to work from home in March of that year and continue to do so even today and for the foreseeable future.

On September 1, 2020 posted a list disaster recovery and preparedness process and infrastructure improvements we'd make over the past year such as:

1. Cloud-based databases running in multiple regions, to better place the system geographically near the library
2. Incremental transaction data backups happen every 10 minutes
3. Database backups are saved in the local region, as well as to a separate region. If a database hosted on the east coast has a disaster, there is a copy of the database backup in another region.
4. Database backups are saved to the local server, as well as copied to S3 storage
5. Database backups are also copied from Amazon's data centers to Google Storage
6. A new automated system restoration process which had been manual prior to the beginning of 2020. As an automated process, it takes about an hour while the previous manual process took 10-12 hours at a minimum.

2019 Updates & Improvements

On August 15, 2019, we shared the news of some additional steps we'd taken to further enhance our disaster preparedness and recovery efforts including:

1. Back-end changes to KLAS 7.7 to it quicker and easier to create and store database back-ups
2. A move to storing back-ups in the cloud, so they are safe and retrievable no matter where disaster strikes
3. A new monitoring app so staff can be notified right away if something goes wrong with the servers and any emergencies can be dealt with as quickly as possible

2018 Updates & Improvements

Our September 11, 2018 our first Emergency Recovery & Disaster Preparedness Key Notes Blog post was written as we faced the threat of Hurricane Florence and shared details about we'd implemented at that point including a combination of both procedural and physical preparedness such as:

1. A gas-powered generator at our office
2. Redundant internet providers, firewalls, and network routers
3. Daily backups of data to our on-site servers
4. Weekly data backups stored offsite
5. Encrypted database backups on AWS S3
6. VOIP Telephone system to allow staff to work remotely
7. Keystone Status Page to communicate database availability, even if we’re unreachable
8. Contingency plans and equipment needed for remote database and customer support

Last week we updated you on how our Washington, New Mexico, and Kansas talking book libraries are faring as their staff and patrons continue to face the ongoing challenges of keeping themselves healthy and safe while operating during COVID-19. Today we want to bring you a similar update from Keystone. 

Since mid-March our staff has primarily been working from home and continues to do so. This would not have been possible without using our established VPN and VOIP services. These have allowed us to access necessary documents and information and continue seamlessly answering customer phone calls. We hope you have not seen a dramatic change in our customer support or development process other than the occasional dog bark or child interruption which I am sure we've all come to expect at this point. If you've had any degradation in service of any kind, please let us know ASAP so we can address it.

Also, this was the first year without a physical KLAS Users' Conference and we sincerely hope the online programming brought to you by your fellow KLAS Users and our Keystone staff have helped to keep you up to date on the latest happenings with Keystone and KLAS as well as continue to hear what you're fellow KLAS Users are up to. Speaking of these ongoing efforts, our next KLAS Users Roundtable will be held at 3:00 PM EDT September 17 and will focus on Virtual Programming for Patrons. More info will be forthcoming shortly. Keep an eye on the KLASusers listserv and our Webinars & Roundtables article for more info.

A new initiative we've undertaken is to move our planned September KLAS Administrators' Training online. For the first time, we will be offering this training for your staff members to be able to attend remotely, eliminating the need to travel to our office. We're revamping the schedule, rethinking the sessions offered, and talking through the best way to conduct this training so attendees can fully benefit from it. This first session will be kept to a small number of participants, so we can try to encourage the discussions and dialog that have made face-to-face Administrators’ Training so successful. The price per person for this online admin training is $600.00, but there obviously are no travel costs to incur. Please let us know if you may be interested in attending a future online KLAS KLAS Administrator Training session.

As many of you will be aware, Hurricane Florence is forecast to bring heavy rains and high winds to central North Carolina, including the area where Keystone's offices are located. We are monitoring the storm closely, and are making plans to minimize the impact the storm may have on us all.

1. We have a generator for critical functions of the office. In the event that our electric power goes out, our systems, servers, network, email, and phone system should remain unaffected.
2. If the roads are unsafe for staff to travel to the office, our phones are VOIP and our customer support staff all have work-issued laptops. This would allow customer support staff to work from home over a VPN and keep supporting you all.
   Depending on the severity of the storm, we may have difficulty responding to support calls as quickly as we'd like, but we'll be doing the best we possibly can.
3. Customers that have their database hosted with us-- for years, we have been storing backup copies of everyone's databases off-site (in encrypted form) in case of emergency. In case the hurricane is really terrible in this area, we have an emergency operations site prepared to be able to bring up all the hosted customers on cloud-based servers within a number of hours. Should this be necessary, connecting to those cloud servers will require local admin rights to load a new set of KLAS support files. We will be sending out information about the backup connections and plans to affected customers as needed.

Another area that we have been seeking to improve: providing a consistent place for customers to be able to check on the status of their database and connections so that it is quicker to identify where an issue might be occurring. As part of the 7.7 release that we are in the final stages of testing, we have been planning to roll out new status pages to report on the status of each library's database.

For this weather event, Keystone is going to use an early version of this system to provide information to all customers on how we are weathering the storm. The Keystone Status notification site is now accessible from the "Keystone Status" menu item on klausers.com and at this URL: https://uptime.statuscake.com/?TestID=emlREBtN3e

This site will show the status of our Support and Hosting networks, and also provide us with a location to post any announcements. The Keystone Support network indicates, among other things, whether our email server is still receiving emails, and if we can expect our phone system to work. The Keystone Hosted Customers ISP indicates whether our hosting network is still communicating with the Internet, and therefore should the customer expect to be able to connect to the database with the staff functions.

In this week's blog post we're pulling back the curtain on part of the software development process called "Regression Testing." It's definitely not the most glamorous part of getting a major new release sent out. In fact, it can be incredibly tedious, but even so, it's a critical part of the process. So, let's talk about what it involves, why you should care about it, and why we're doing it right now.

Why are we currently doing Regression Testing?

KLAS is built on Progress Software's OpenEdge Application development platform. KLAS 7.7 was created on OpenEdge 11. Upgrading from one version of OpenEdge to the next includes significant changes and additional features and functionality our development can leverage in future KLAS releases. For example, OpenEdge 12 will allow us to support KLAS on Windows 11. Any major upgrade to the OpenEdge platform necessitates a major release in KLAS because it fundamentally changes KLAS' code base. So, this translates into needing to make sure upgrading from OpenEdge 11 to OpenEdge 12 doesn't affect how KLAS currently operates. 

Additionally, a major operating system change (such was Windows 10 to 11) can also have unpredictable effects on how KLAS runs. As such, we are regression testing in several Windows environments: Windows 10, Windows 11 prior to version 22H2, and the latest version of Windows 11.

How does Regression Testing happen?

Essentially, Regression testing is the process by which our staff test every feature, button, and process within KLAS after a major shift in the underlying KLAS platform to ensure it still works as intended. One thing to keep in mind is that some modules of KLAS have different versions for our different types of customers (IRC, LBPD, and PNDB), so each of those versions has to be thoroughly tested. Integrations and web services need to be tested to ensure that KLAS still communicates correctly with PIMMS, Gutenberg, Rolka-Loube, and even just our own WebOPAC and WebOrder systems. Finally, we need to check that nothing has broken for those using KLAS with a screen reader.

So, during Regression Testing each module, tab, and function is put through its paces to make sure it works and anything unexpected is documented. Here's how that happens:

1. A copy is made of a test database (originally based on customer data) representing each version of KLAS: LBPD, IRC, and PNDB.
2. The new test databases are upgraded to the new version.
3. In this case, Virtual Machines are created running the latest version of Windows 11, and the same test databases installed on the VMs. 
4. A spreadsheet is created of all modules and major components in KLAS. 
5. All Keystone staff that are helping with testing use the spreadsheet to record their process as they work through through each screen and function they're assigned to test, recording what has been tested and in which version of Windows.
6. Notes are made about any issues that are discovered, so they can be addressed and re-tested prior to sending the release out to our beta site customers.

What does this actually mean for me?

It means we've completed the OpenEdge 12 upgrade in our development environment, are making sure it is stable on both Windows 10 and Windows 11. Upon successfully completing regression testing, we will release KLAS v7.8 to our beta site customers. If they report no major issues or bugs that we missed despite these efforts, KLAS 7.8 will then go to general release.

Because of the countless ways KLAS can be configured, and the extensive possible combinations between customer version, Windows version, web browsers, printer drivers and so forth, it is impossible to test literally every process in the exact environment you will run it in. However, we are doing our best to be as thorough as possible, so you can be confident that the upgrade will not disrupt your operations.