Best practices for restricting developer access to UAT and production environments, yet still getting anything done

Question

I am working with a small company, that is going through the growing pains of transitioning from a start-up culture to a more mature corporate culture. In the past, developers have had more or less free reign to access UAT environments, and even wide latitude to access production.

However, under the new approach, developers have access only to the Dev and initial QA environments... and are locked out of UAT and production. All access to those environments, from deploying code (Java WAR's in this case), to managing Java app servers, to even reviewing logs and the database, has to funnel through a sysadmin.

It's still early on, but so far this approach does not appear tenable. The net result is that every time there is a production issue, or even just a bug ticket entered in UAT... it requires an "all hands" meeting, with half the department crammed into someone's office or huddled around one person's monitor.

I would like to propose something more workable, while still satisfying the need to restrict access to sensitive user data, etc. One idea that comes to mind is to create a readonly-mount for the log file directory, at some other location where developers may at least view the application-level logs. However, beyond that I am interested in best practices for how to restrict developer access while taking the smallest productivity hit possible.

NOTE: I did find some vaguely-similar questions before I wrote this one. However, they were either too narrow (and here), or dealt with Sarbanes-Oxley matters not at issue here, or simply asked "are these restrictions normal?" rather than asking how to cope with them.

Answer 1

We made that change in the last few years here. We however allow most developers access to production logs and read access to the database. We simply don't allow them to change anything directly on production. All changes must be code reviewed and code is pushed first to QA for testing and then to prod by either configuration management or the technical leads.

We also frequently refresh our dev environments with prod data. Since all dev changes to the databse and to database data like lookup tables are scripted and storeded in source control, it is not hard to refresh from prod and then quickly load all the dev changes. Since a good number of the reported issues have to do with the data that is in prod for one or more particular records, this helps us to see a more accurate view of the prod data. Having a separate dev server for project development and one for support of existing prod issues could work too. That way you don't mess up the people working on project work by refreshing the data base to work on a bug reported from prod. Hardware is cheap compared to the cost of a bunch of people sitting in one room looking at one screen to try to fix a problem. It probably take avoiding only two or three of those meetings to justify the cost of a server that can be easily refreshed with prod data.

Our users are aslo well-trained to provide screen shots. Amazing how many things can be fixed by seeing the screen shot. Sometimes the fix is easy (the user is looking at the wrong screen) and sometimes you can see that the record has data you didn't think it could have or lacks data it should have. (Who changed that field from not nullable to nullable? No wonder the code is broken.)

We also use our audit tables extensively to find where the problem might be. We store what application made the data changes as well as the change since we have many possible applications hitting the same database. It is helpful being able to see exactly who or what changed the dat to it's current incorrect state. That helps us find where to start to fix the problem very quickly.

If the data is restricted (we have developers in another country who cannot see prod at all), the it is up to the tech lead to do the research to find what is causing the problem and then assign the fix to a developer after providing him with details from his research. This makes one person responsible for the research, not the whole team and devs don't get involved until after the intial research is done.

It wasn't an easy change culturally. People rsisted at first and hated losing their prod rights. But after awhile people came to grateful that they didn't have to make changes directly on prod (and get whacked if something went wrong! Oops forgot to highlight the where clause and deleted the whole user table.) We were grateful that we had the chance for code review and QA testing (which we also didn't have when devs had prod rights). We saw first hand that there were fewer crisies created by a bad push to prod under pressure or by doing something directly on prod and forgetting to do it to dev to (or put in source control), so those times a later push from dev overwrote a change on prod went away. And we were oh so glad that no one was deleting the whole user table by accident (thank god for audit tables) anymore.

Answer 2

Well, here are some thoughts

• Have the log files exposed as read only. Make them available on request, or serve them up via HTTP to a page so developers can get at them when they need them. If you make sure you don't log customer information to the logs, that makes it easier to get this agreed upon.
• Personally I don't think it is a bad thing to allow developers read only access to the application server console screens (for example a WebSphere Application Server). This way, you can view settings. Passwords and things should be hidden anyway, so it isn't like you can steal secrets. Plus, by view only, you can see if something is amiss, and always request higher up sysadmins to make the changes needed.
• If your company has the time to do it, it would be worth classifying your data. What is customer-based? What is data that may be the "secret" to how your company does business versus other competitors. If you classify your data, you can now better define who can have what access to what. Even in production, the DBA's could set up some developer user ids that allow you select (read) access on the classifications of data that are ok to look at. Anything that needs a clearance, you may have to go to the DBA for, but...you can at least query for some things without needing the whole "war room" to do so.
• If you afford to build and maintain it, have a mirror of production somewhere. This mirror is configured exactly the same as production. It has the most recently deployed version of the application there. And if you need production data, you take a back up of the data from production, run it through a "scrubber" or "scrambler" if you need to so satisfy your execs, and the upload it to this mirror. Voila! You now have as close to production as possible, but you are allowed to have more authority in this environment to troubleshoot. This will give you everything except exact production data (since it was scrambled). If the bug turns out to be something along the lines of specific data, then you go back to querying production and/or working with the DBA on it.
• Also, perhaps it would be worthwhile to have some profiles on standby? Perhaps your sysadmins could grant you some temporary authorities to things? This could be argued as good or bad either way and is ultimately a company policy.
• Also having a good change management system in place goes a long way to identifying candidates for what just went wrong. If your application suddenly crashes, it would be helpful to know that the sysadmins just applied an OS patch to production and this OS patch may in fact be affecting your application. Or maybe somebody else moved a fix to production and didn't fully test it. But you at least know what could be potential candidates for breaking things.

Ultimately, all of these things or others, are things that your company and your IS/IT department need to decide. Perhaps this is something to discuss with your CIO? Or your manager? (Of course it depends on how well they accept feedback and comments. Where I work, I'm very lucky that they appreciate those kinds of thoughts from us.) Perhaps, now is the time to begin to discuss these things.

Also it doesn't hurt to present the cost of the risks versus the benefits. If a temporary grant of authority to the logs, the db, etc. gets you "up" in production, perhaps that risk, is outweighed by the satisfaction felt by your customers, both internal and external.

Anyway, just some thoughts to think about. I'm not sure if there is a right or wrong here. It is just something every company has to make their own and then live with their choices. But kudos to you for getting the ball rolling on this conversation.