Chef Monitoring

 At Schuberg Philis we have a certain way of working which gives every engineer a lot of freedom. That amount of freedom can only be provided and maintained with the right tool-set and auditing capabilities. Auditability and accountability are the main reasons why we need to have our change information out in the open and visible for every engineer. Having that information out in the open enables us to peer review the changes made. No secrets here and no need to hide any information, at the end of the day we are all working towards the same goal. The last couple of years we started using Chef Enterprise, which we love using btw. The current out of the box Chef Enterprise solution is however lacking some auditing capabilities that are a requirement to maintain our way of working. To resolve the lacking auditing capabilities we created Chef Monitor, it provides us with the change information in a format we like and enables us to maintain and scale our way of working while adhering to our strict requirements. 

Chef Enterprise

Chef Enterprise is the central place for all our system-configuration. It is however hard to see all the changes that are being made, so Chef is sometimes perceived as a “black box” controlling all configurations within the enterprise. If configured right, the amount of control is awesome and saves you a lot of time. But with 150+ engineers changing cookbooks, roles, node properties or environment settings, you might want to know who is changing what! Where the hell did this attribute change? Who updated this role information?. Unfortunately there is only one source you can use to retrieve this information. The source of change information is the nginx log on the frontend of your web server. Splunk, Graylog, Kibana or another tool could help you to see that a role or cookbook has been touched, but you still miss the actual change information itself.  

Chef Monitor

With the knowledge that tracing changes in Chef is not trivial in mind, we decided to create a processes to eliminate the problem of change tracking. As a result Chef-monitor was created. The monitor is built out of two components: “chef-logmon”; a process running on every front-end server scanning the nginx log and sending this information to the existing RabbitMQ service. The RabbitMQ service is the actual service that lives on the back-end that comes with Chef Enterprise by default. On the back-end a component called “chef-worker” is running, which picks up all the information from RabbitMQ. The “chef-worker” downloads the object out of Chef Enterprise and stores this on disk, this process can run on a separate “monitoring server” and it only needs access to Chef with an account that has enough permissions to read every object. The “chef-worker” is combined with a SVN or GIT repository, sending a diff on every commit, showing you exactly what has been changed on the object. The repository can also serve as a backup of your entire Chef environment, with just a few lines of code you can import everything into a new environment. Implementing this in a typical high available Chef Enterprise environment, will architecturally look as follows: 2 front-end web-servers (in blue), 2 back-end database servers (in red), adding our monitor server (in green)  


chef-highlevel


Having all the required objects in place, as described above, the process will look like this:
 

 monitor-process

  Scanning the log and downloading the object can take a couple of seconds. A situation that two people change the same object at the exact same moment can occur. There is a possibility that you miss one of the changes when there is a simultaneous change, this is however apparent in the diff which makes troubleshooting a lot easier. The result of a diff looks somewhat like this:


commit  

Open Source

If you want to know more about this, our cookbook and chef-monitor gem are open source. Use the links below to read more about it or to implement this yourself. Contributions in any kind or form are also highly appreciated.

Cookbook: https://github.com/schubergphilis/chef-monitor-cookbook

Gem: https://github.com/schubergphilis/chef-monitor-gem    

7 Comments

johnbellone
RT @_Harm_B_: Great blog by @sanderbotman on peer reviewing code in #chef @getchefdotcom http://t.co/0vZ8D6oDvH #devops http://t.co/1vKuElD…
julian_dunn
The folks from @SchubergPhilis released Chef Monitor earlier this week, an audit logger for Enterprise Chef. Wicked! http://t.co/lB0hWpRNMQ
dpnl87
RT @julian_dunn: The folks from @SchubergPhilis released Chef Monitor earlier this week, an audit logger for Enterprise Chef. Wicked! http:…
skeptomai
RT @julian_dunn: The folks from @SchubergPhilis released Chef Monitor earlier this week, an audit logger for Enterprise Chef. Wicked! http:…
nulleric
RT @julian_dunn: The folks from @SchubergPhilis released Chef Monitor earlier this week, an audit logger for Enterprise Chef. Wicked! http:…
rodeeend
RT @julian_dunn: The folks from @SchubergPhilis released Chef Monitor earlier this week, an audit logger for Enterprise Chef. Wicked! http:…
TonyNotto
RT @julian_dunn: The folks from @SchubergPhilis released Chef Monitor earlier this week, an audit logger for Enterprise Chef. Wicked! http:…

Not Published

0/1000 characters
Go Top