Every one hour a backup of our SQL data is created. This data allows for recovery of websites, and ALL application data and data entry. This covers our own administrative system as well as all things relating to mediate.com and caseloadmanager.com (plus stand-alone client domain websites).
Every two hours a backup of ALL user hard data is created (such as pdf, image files, media files and EVERYTHING else not stored in the above SQL databases). The idea here is that most of the time, this is data that the user can recover themselves, such as image files, media files, etc., and so there is a copy already in existance (on the user's PC for example).
Both the above (SQL and user data) backups are stored in two places: on the live server itself (easy and quick to access if needed) AND on external S3 storage (in the highly unlikely event that we lose access to the backup drive on the live server).
We keep 7 days minimum of both of these backup types. As the old backups are removed, they are first archived as weekly (4 weeks minimum), then as monthly (12 months minimum) and then as annual (no limit at this point).
In addition, weekly backups of the ENTIRE server (all servers) is made in two different fashions, both at the Amazon Web Services (AWS) level. One is called AMI backups (AMI = Amazon Machine Image). The other is as AWS "snapshots" of all the drives, in their entirety. A minimum of the prior FOUR weeks are here stored.
In addition, monthly backups of the ENTIRE server (all servers) is made in two different fashions, both at the AWS level. One is called AMI backups (AMI = Amazon Machine Image). The other is as AWS "snapshots" of all the drives, in their entirety. A minimum of the prior TWELVE months are stored.
In addition, annual backups of the ENTIRE server (all servers) is made in two different fashions, both at the AWS level. One is called AMI backups (AMI = Amazon Machine Image). The other is as AWS "snapshots" of all the drives, in their entirety. (no limit at this point).
These weekly/monthly/annual "entire server" backups are kept in both our USA "home region" with Amazon, and for redundancy in case of a total regional meltdown (which Amazon claims is near impossible), we store these "entire server" backups in a European region as well.
What happens in an emergency?
It depends on the definition of "emergency." If the emergency is client centric, the client should STOP working on data entry until the issue is resolved or fixed. This would be emergencies such as "our website is crashed," "our data has suddenly (and for no apparent reason) gone missing," "our data seems to be corrupted," etc.
To clarify a bit further: "I accidently deleted all my data entry work that I entered this morning" is not an emergency. It is a "problem" caused by operator error. (The above three examples are not caused by operator error). Not to say that we are unwilling, or unable, to deal with "problems" caused by operator error. We can try to fix operator error type problems too! But, as an example of a "problem" that may not be fixable: if an operator losses work because (s)he accidently deleted it, then that may or may not be recoverable. Remember, our data entry backups (SQL) are done in a one hour granularity. So, if the backup was made at 9am and the operator enters data from 9:05 to 9:50, then accidentally deletes it at 9:55, we won't be able to recover it. BUT if they enter at 9:05 to 9:50, then accidently delete it at 10:01, then we can recover all the data (since a backup is again made at 10am (once per hour), shortly before the operator deletes it. Thus, operator caused "problems" may or may not be fully or partially recoverable.
What should I do when I realize I have deleted my data?
Call the RIS number (541-345-1629) and inform the office that this is a VERY important issue (data problem or possibly emergency).
How quickly do I need to contact you if I have deleted my data?
Contact us ASAP, assuming that you wish to resume data entry ASAP. Once you start overlaying new data on top of the database AFTER you accidently delete info, then recovery of the deleted data becomes vastly more complex since data would need to be merged (the new data kept, and the data they deleted restored AND MERGED IN with the data they entered after the accidental deletion. Not only might this NOT be possible at all, but if it is, then the time from our staff to do this goes WAY up as well as the task becoming vastly more complex).
To clarify a bit more: IF an operator accidently deletes data entry work, let's say for example at 3:10pm on a Tuesday, they are FREE TO WAIT up to a WEEK before contacting us and asking to have the deleted data restored (and in this example, we can take them back to 3:00pm Tuesday using our backups). BUT it is important that they stop further data entry until they get the problem fixed (see above paragraph). Of course, they are free to re-enter their deleted work, and proceed that way (without needing to call us at all, obviously).
What is the cost?
RIS' charges "by the hour" for recovery of operator caused problems, at a cost of $175/hr. If it is a "emergency" (as per earlier in this document), then of course they would not be charged (unless it is determined that the data loss was their fault).
ONE MORE THING worth mentioning: In these examples, we use round hours. BUT the hourly SQL backups are actually made every hour at FIVE MINUTES past the hour (not at the top of the hour) and the user data is made every two hours at FIFTY TWO minutes past every other hour. (This is so the various backup types don't conflict with each other).