Change Mgmt 2263

  • I have a situation where changes are being made right in Production without testing or approval prior to executing the changes in Prod.
    I have 2 questions:

    1. IT said they don’t have a test environment (they are using QAD). My question is, does anyone know if QAD would allow you to create a non-Prod environment/instance?
    2. How have you seen this situation handled and/or remediated?
      My ideal approach would be to:
    3. Get them set up with a non-Prod environment where it gets refreshed from Prod frequently enough to test patches or fixes recommended by the Vendor.
    4. Demonstrate explicit approvals from the business prior to making the changes or executing scripts affecting the data/database in Prod (requests aren’t implicit approval, I am sorry…)
      Any insights you can share would be greatly appreciated. Thanks in advance.

  • Hi SG - Yes, I agree with you, as ‘direct changes to PROD’ fails general IT control principles. If I were a SOX auditor I’d certainly frown on this and it increases the risk for accidents and even worse. This violates COBIT 4 which is often used as a checklist for SOX 404 compliancy by external auditors.
    The only direct changes to production that should made to PROD, should be in an emergency situation with a ‘fire ID’, extensive logging, and releasing the change the next day through normal change management procedures (e.g., programmer called after hours to fix an abend).
    While the QAD environment (often called UAT or ‘user acceptance testing’) might be considered ‘test’, it should simply be a staging area where developers move their source objects into for promotion by an ‘authorized releaser’ into production.
    I’d suggest the following:

    1. Setup TEST, QAD, STAGE and PROD environments for all applications (STAGE is just a holding area for release candidates)
    2. Establish a list of authorized releasers (e.g., supervisors, managers, or senior IT developers)
    3. Ensure no one person can promote ‘their own’ work from STAGE to PROD (e.g. an authorized releaser making changes should have another one release it)
    4. Setup an emergency release system for after hours support
    5. Use network security controls so that IT developers can read PROD but not update, only Authorized Releasers can do both.
    6. Use email or Intranet based web pages to briefly document changes as well, so you have history and a good audit trail.
    7. Monintor and tweak the system so that IT releases are efficient. Also, pilot test and move into this gradually system-by-system or server-by-server, so that IT is not disrupted too extensively. You may also need management backing if you encounter resistance.
      Good luck 🙂

  • Thanks, Harry, for your feedback: I knew I was not off base but the IT Manager AND the CFO keep dimissing my concerns and recommendations.
    While they said they have a test database but the IT manager said they only have 1 application program and keep insisting that changes must be made in Prod. I was very confused as to why there was a disconnect… I have since suggested to include the external auditor in the discussion so they can hear it directly from the external auditor (who will share the same view as mine, I am sure). Sometimes, that’s the extra push and support we need from the external auditor for Mgmt to listen…
    Thanks again for your reply. I appreciate that greatly…

  • Have heard this earlier. No Quality or test environment to simulate the prod environment. Cant afford, Cant create, technically not possible…the argument varies but still is there. In most extreme case try this:

    1. Full backup of the application, database and OS before the change implementation in production (This is the entry criteria for change implementation for each application related change). This is a special category backup with no rotation of tapes for a year.
    2. Well documented roll back plans must with each change. They needed to be approved with the change request.
    3. Roll back window where business user approvals are sought for the fact that changes have been implemented alright. Changes to be signed off as success by IT only after that.
      Idea is to increase the overhead associated with each change so that people see value in implementing the test environment. This helps apart from the external auditor buy in.

Log in to reply