Posted by Rory Scott
As a manager, executive or business owner, would you consider not requiring your employees to track their time? The most likely answer is a resounding, “No,” even for exempt employees. The reasons for tracking employee time are numerous; a few might include payroll (for hourly employees), staffing needs, employee accountability, as well as profitability through profit and loss statements. While the benefits of employee time tracking are widely understood, the same level of accountability and tracking is often not applied to internal technology and data systems and processes.
A properly implemented logging system will give your business insight into the technology, server-side and data systems and processes currently employed. The reasons for a logging your non-human resources are almost as numerous as tracking your employee time, including but not limited to any errors encountered, the time each process takes, and the resources each process consumes. Additionally, the time-savings when something does break is extremely beneficial to your technology/IT teams because it gives them specific details about the problem that would be lost if no logging system was in place.
When designing, developing or deciding which logging system to use, it is important to keep your specific needs, your current resources (and their level of troubleshooting ability) and growth/scalability in mind. At a minimum, you should have easily at hand the number of processes and programs in production, the time each takes to complete, and the overall status of each. For example, there might be five data sourcing programs that run every morning starting at 4:00, there are three reporting processes that start at 5:30, and one email program that runs upon completion (success or failure) of the prior programs and processes. If one fails, your team should be able to identify which one, the reason why it failed, when it failed, and what other processes its failure impacted. Without logging, the team would have to perform expensive manual testing to determine which process failed, why it happened (was it a bad function, improper formatting, bad dependency, API downtime, the database is down, etc.) and when. Without monitoring, it is impossible to know when the failure occurred other than to manually run the process or check that the process did what it was supposed to do every time it runs.
A basic way to approach this solution that will cover many purposes is to have a logging process with two levels; the first is a high-level log that quickly tells the analyst which processes ran, when they ran, how long they took to run, the filename associated with the process, and a brief message describing the process. Additionally, there is a detail-level log that captures specific processes happening within the high-level log. This log can be ignored unless there is a failure in the high-level log, making daily monitoring quick and easy. In the event of a failure, the analyst can quickly identify the parent entry in the high-level log, see any processes and files associated with the failed process, the specific error message, the client to whom the error applies, and the date, giving specificity to the troubleshooting process.
Tracking your internal data processes and programs should be a priority similar to tracking employee time. Many programs and languages come standard with some form of logging (nohup.out can be used as a basic log in Linux, the logging module works well in Python, Visual Studio can be used to create a log in Windows, to name a few). Other options include building a custom solution, which might be as simple as creating a log table in an SQL database with start times, stop times, and program names. While not a sexy part of programming, it quickly becomes necessary as more programs and processes are added so that manual intervention and validation can be excused for process monitoring.