Fundamental Segregation of Duties 320



  • Brett;
    Sounds reasonable, a few questions come to mind though.

    1. What mechanism prevents a developer from moving code or executables to production outside of your version control?
    2. How often would the automatic comparison of code and executables be run? And what would be the procedure followed in the event a difference was discovered?
    3. How solid is your record of who actually accessed the production system to implement a change? Can the audit trail be erased or altered by anyone?
      A few things to think about.


  • Sounds reasonable, a few questions come to mind though.

    1. What mechanism prevents a developer from moving code or executables to production outside of your version control?

    Nothing. Of course, nothing would prevent whoever does the move to production from substituting different code. That’s why it would seem to be a good solution to be able to verify at any point that the production code matches the test code and matches the dev code. I think this solution is would be desirable even if we had separate developer and application release roles.

    1. How often would the automatic comparison of code and executables be run? And what would be the procedure followed in the event a difference was discovered?

    Good point, I’ll need to make sure the procedures cover that.

    1. How solid is your record of who actually accessed the production system to implement a change? Can the audit trail be erased or altered by anyone?

    We will have event logs of who logged in (this is a Win2K server). If we configure security properly, the person releasing to production should not be able to modify or erase the event log. Of course, the hardware/OS administrator can always erase the event log, but there is no way to avoid this possibility on a Windows server, as far as I know.
    Thanks for pointing this out, I’ll need to make sure this is the way it is configured.



  • All,
    I’ve been following the thread and have some questions/comments. I’m in a small shop of two, manager and myself. I’ve been requested to make my documentation SOX compliant. I’ve been visiting site like this one to get an idea of what that means. For example I have written several commission systems. How do I make documentation SOX compliant?
    From read this thread and others I’m getting the understanding that the end user of the system would need to add verbiage to the document stating how they tested the application and the level of accuracy.
    Would appreciate any insight
    Thanks,
    RC



  • I just left a client that had the same issue with segregation of duties as regards the implementing and making changes to production systems.
    Due to the small size of the IT staff we all agreed that if the business puts in place a mitagation that will consist of the review of any changes that are made by anyone in the IT department then they would pass. The IT department then created a daily exception report that list anyone who had made a change to any of the production code. The report is generated each day and is forwarded to the VP of IT and the CFO for review.
    Let’s face it not all IT departments are going to have 100 people running the operations. Sometimes as an auditor you have to work with the client in finding solutions. Not nail them and force them to make changes that will reduce the overall revenue of the client.



  • Sarboxian05,
    Due to the small size of the IT staff we all agreed that if the business puts in place a mitigation that will consist of the review of any changes that are made by anyone in the IT department then they would pass.
    Does that mean creating an online document that the IT Director would look at everyday and approve or not the production modifications? Lets say the IT modification/solution is what the end user wants but for what ever reason it doesn’t pass the IT director smell test for SOX (what will he be looking for?).
    What is the ultimate goal in being SOX compliant from an IT perspective?
    Any insight would be greatly appreciatated



  • In this case it was not a user request but a remediation of the problem of the IT group as a whole having access to production source code due to there being such a small IT staff. If one person was out or on vacation someone else in the IT staff can make changes to production. The request for a way of monitoring the production code came from the External Auditors. The client was able to create a ‘job’ that basically scanned all the production source code and when it found any code that was changed it generated a report with the code that was changed and the associated user ID that changed the code.
    The most important part of the remediation was that the report is reviewed on a daily basis by both the VP of IT and the CFO of the corporation. It is a manual process that forces the Management to verify no changes are made to the systems without authorization. The alternate solution was to hire another 2 bodies that will a) work in the dev environment and make changes b) someone to review the changes and move them to the production environment c) Then the orginal person to review the production environment and verify that the move was made correctly and run the source code.
    The point I am making is with a small IT staff there are times when you can’t hire more people so you have to come up with alternative solutions to address segregation of duties. After all when we hire people we hire them to do a job and we trust that they will do that job to the best of their ability. Sometimes you lose and get a bad apple but most of the time employees are trust worthy.



  • I am working with a small team: a manager and one support analyst/developer for each of three legacy systems.
    We are struggling with SOD issues our main mitigating control is based on a report showing latest dates for programme changes being compared against HelpDesk tickets that should match by date and show authorisation for the changes.
    We have real issues with one system where the developer is required to make changes to production data to resolve user problems, the application has no usable audit trail - I have proposed that we use business controls in mitigation.
    Any comments would be appreciated.



  • At a client with a small IT shop, our solution was that the developers would not normally have access to production.
    The policy says that when a change is moved into production, if the implementor (developer, contractor, vendor, etc.) does not have access, they will be given temporary access to implemnt the change.
    After the change is verified via testing, the implementor’s administrative rights on the production system are removed.
    And of course for emergency changes, they can also be given access.
    This seemed like a good solution, what doesn’t it cover?
    Also, they use a free file integrity tool to monitor for unauthorized changes.
    marc



  • Wow, I wish I’d discovered this forum months ago. Excellent discussion - I especially like the idea proposed by the poster with the automated comparison of production objects to the versioned objects.
    Our IT shop is not really small (around 60-70 developers), but the same issues exist. Our operating model is that of an operationally exellent company, not much for bells and whistles, execution is the key, and avoiding processes that don’t add value. Our IT directors, and I can’t say I disagree with the principle, contend that the developers’ expertise with the systems and how they relate with the database make them the most qualified to troubleshoot production issues, so the majority of them split time to do support when the failure is in the systems they are most familiar with.
    None of these developers would look fondly on being relegated to a support-only role (locked out of development), and they don’t plan to staff that function with dedicated technicians. The result is that the SOD issue in IT is the control deficiency causing the most concern with our internal and external auditors.
    Honestly, the developers prefer to move code through the change management system. The system gives them additional assurance that objects are placed in the correct environments, at the right levels, etc. etc. - our auditors are more concerned with their ability to modify data in the production environment, which is often a necessary step in resolving a design problem in production that has already updated something incorrectly.
    Of course, this does give them the ability to do all sorts of malicious things - but they all realize the net result of doing something instantly destructive, like dropping a major database table, would be about the same as coming to work in their pajamas - they’d be sent home immediately, probably for good. The concern seems to be that they might use their in-depth knowledge of the system to do something more devious and harder to detect, which might require a combination of data manipulation and some back-door programming in the system to act on (or ignore) the foreign data. Other than to commit fraud, there would again be no real point to doing this, and I would imagine most of these developers would know better than to try, as there are layer upon layer of sound mitigating controls in the finance areas that would reveal anything significant in short order.
    We’re still searching for that happy medium. I could argue all day that even our review/approval process for manual database changes by IT (performed by someone outside the development group) doesn’t provide much extra assurance, since it would be hard for anyone without the same in-depth system knowledge to assess the impact of a particular change running all the way through the system, especially since the output would ultimately end up in the hands of a business expert in whatever financial area were impacted. It’s just not the same as a single A/P employee having access to vendor setup, invoice and voucher entry, and the key to the check room - for numerous reasons. Yet - for some reason it seems to scare the crap out of them (the auditors, not so much the business managers).
    I realize this isn’t a question - but I’d love to wake this thread up if anybody has any more wisdom to add. I just know that audit is never going to let this thing go - even though we’ve done our best to implement mitigating controls, both automated and manual. They seem to believe the physical segregation of staff is the only way to strengthen the control, and I don’t see the business forcing this change to happen. Will I ever get this monkey off my back?.?.
    Thanks for letting me vent… be well, all. You have a nice community here.



  • I’m struggling with the same issue. We have a group of two people doing all the develoment and migration to production with no QA or UAT environment, just dev and prod. I read somewhere that without true sOD ‘the developer must log every activity they perform on every production system ( and end user desktops running software count as production systems), and a manager has to validate it’s authenticity’ (from blogs/ittoolbox.com/visualbasic/dotnet/archives/001029.asp) but I’d love to get this statement from a more authoritative source. I guess I’ll have to ask our in-hous auditors for an opinion.



  • You may also want to more pro-actively identify changes to production source / object code through using some form of change detection system such as tripwire. This would alert on changes made directly to production.



  • Great thread. Glad I stumbled across this today. If anyone can help me with this, it’d be greatly appreciated:
    We’re facing a similar issue at my client where SAP support staff are making master data changes (t-codes, cost centers…etc) in production directly. BTW, these folks are developers as well. They’re following a change management system to control development and they have competent administration staff to provide production transports.
    However, there just isn’t anything they can do, from an operating standpoint, to stop IT support staff access to master data, as this would disrupt business greatly.
    We’ve thought about logging their activities and having internal audit reviews on a consistent basis to ensure proper approval was granted, but this may not be feasible in SAP. Does anyone know if specific changes to master data configurations can be logged through SAP, Oracle or through a Windows-based server?



  • One thing I’d like to add outside of my question above is that overall it’s important to consider the financial impacts of each potential issue. Auditors need to understand that there is no one-size-fits-all, checkbox list of controls for IT. This is partially due to the inexperience of external auditors, either because they are fresh college grads or accountants who don’t understand IT operations.
    Since the potential damages caused by control deficiencies in IT, especially for SOD, are typically inferred, challenge your auditors by first considering what the ultimate outcome/what could go wrong (risk) of the current set-up is, and what it is YOU do to ensure the relied-upon outcome is secure and produces accurate outputs for financial purposes. When I say relied-upon outcome, I am referring to any financial reports or data that is relied upon to generate the financials. In the end, its the accuracy of the numbers that matters, not who has access to production.



  • I am a consultant for a large global bank, but my group is only 5 people. We are being hammered to comply to this act with no regards for how thin our staff is. The basic problem is isolating development, qa, uat, and production.
    The auditors are claiming that developers cannot have access to qa, uat or production. They are also demanding that there will be some kind of seperation of source control such that migrating source from dev to qa requires physicaly moving the source code somewhere. I guess their belief is if the source physically resides somewhere else than somehow that will make it safe.
    The remaining request is to enforce that all developers when requesting source control send some kind of request to senior management. The manager will then have to perform some action to release the code in question to the developer. All of this, I am sure sounds wonderful on paper, and in a power point presentation, but in the tranches, this will create an incredible nightmare. The other problem is that it is a very hard to fight an external auditor that reports to your CEO.
    My final note is that too much power is being given to the auditors. The Act is very vague and from what I gather does not deal with IT at all, but sort of mentiones it in passing. And yet, this act is being used to push forward all kinds of potentially dangerous changes.



  • I completely understand your issue, we have come across precisely the same problem.
    The trick is to (a) consider the risks and (b) consider accountability.
    The risks you need to counter are fraud and business disruption. Audit’s approach to this is that if the developer can’t hack the code in production then all is safe. Obviously this is not the case.
    Taking each point in turm: -
    Segregation of Environments
    What you need to ensure is that if a developer makes an unauthorised change in the production/qa/uat environment then someone knows about it and does something about it.
    Also, if they need access to those environments they must get explicit authorisation preferably each time. Also, it’s a good idea to record their sessions using Keon or some other tool.
    Essentially thisrecognises the fact that for practical reasons your staff need access to those environments, however you have compensatory controls in place to mitigate the risks.
    Promotion of Source Control
    Another one I’ve come across. Audit, as I understand it, want to ensure that the ‘source’ that is in production is immutable and instead of stating the requirement they have requested a solution.
    With most source control systems you can freeze a release i.e. lock down the code that made a release. If you can demonstrate this then the auditor should be satisfied.
    Access to source code
    You need to ask the auditor what the risk is here. Is the code that sensitive that you need authorisation to even look at it?
    Auditor reporting to CEO
    At the end of the day it is the CEO’s decision what controls he wants to put in place - it’s his signature on the 404. It’s the auditor’s job to point out the risks and weaknesses and make recommendations (the latter in my experience can become ‘requirements’). It’s your job to let the CEO know the realistic options from which he can choose.
    Hope that helps
    Mike



  • I respectfully disagree with the earlier gentleman about allowing developers direct access into production. I would agree with you if it were a small ma and pop shop and you were ma or pa and you were the only ones potentially impacted. In a large shop this could be quite dangerous, accountability drops to zero, mass amnesia or finger pointing strikes, and what could be a 10 minute recovery blossoms into hour hours . Segregation of duties is a fundamental control. Yes, it’s slower. Yes, it opens you up to bureaucracy (but lets face it we have to tackle that nightmare anyway). And yes it can be a monumental pain. But to just say I should have complete access from beginning to end is scary. And can be a huge risk. Just because something hasn’t happened to you yet does not mean it couldn’t or won’t.
    If nothing else with seperation of powers it helps ensure an extra set of eyes is involved and exposes someone, whether they are sneaking something in with best of intentions or not, to being exposed by another person. I.e. they are taking more risk by acting in the open and interacting with third parties who could catch them.
    The DBA should not write the application code. The developers should not have direct access to the database. Production support should not be writing the code but should be examining it (dev may be tempted to bypass everything so watch for doors), signing off on it, and handing it over. These groups should be watching over each other. They are a balance to each other. Yes, there is still access and risks involved such as with the sysa but if proper controls are in place you can hopefully keep this to a minimum and be somewhat, at least, prepared if something bad should happen. You may have a small enough shop with two people in it who are the most ethical and wise in the world but to assume everyone is this way or allow unnecessary risk is foolish. If impact is small then hack away but if impact is big and others are involved you have a duty to try to ensure your environment is properly protected.
    I’m not crazy about it either but a higher awareness for us all is a good thing. This process can help you too … suppose you are oncall when one of your 60 developers was moving around doing something and accidentally broke something. He or she could tell you… maybe. If you had some kind of enforceable process you might have a little more information available to you so you can identify what happened and fix it before the business rails on you. Lets be honest too. Usually you have more time developing something than you do fixing a live, or what was once live and the business would like it back again, process. Businesses tend to be a little pushier about the second part.
    P.S. I found the environmental differences comment someone made as a justification for direct access highly entertaining. If you can’t control the testbed or mirror your environments, and are not looking for differences between the environments prior to entering production, all the more reason to bar the gates. Many times that is how environments get out of whack in the first place. Also lets be honest… its one little change and it would be quicker and easier to just put it in instead of messing with the test environment we haven’t kept in sync anyway because we have 60 developers playing around and the business wants it now or I want to leave early for tennis or whatever… so…



  • On relocation of source code to QA…
    If only one person has access to the code and passes it to QA I don’t see how, but don’t know, that makes a difference.
    If a shop is large enough to have 60 developers not a good idea to let final tested code hang around. Once you are satisfied it’s good hand it over to another party to be locked down. It’s possible those people could tamper with it but if their skin in the game is ensuring integrity of the final package they are far less likely to play with it than development. They may also be familiar enough with the ‘thats not what I turned over’ story to have a process in place to prove the original file has not been altered.
    If the company is large enough to pay for 60 developers than it should be able to buy or lease one server for code control and probably has enough in the game to warrant doing so.



  • I’ve looked at the segregation of duties matrix and can I ask please?
    It appears to be mainly driven around financial controls and yes I understand SOX is mainly financial but does it cover conflict of interest in general? Or do ‘we’ only care when a financial application is involved?



  • This is a great conversation, I hope it continues.
    What I am looking for are compensating controls when the segregation of duties has been violated. On occasion, we have a need to have the developer also install the code into production. Obviously this creates an issue, and I need a compensating control to mitigate the impact of this activity. No, I can’t prevent it…Already tried that avenue and failed.
    Does anyone know of sufficient compensating controls to smooth this over with the auditors? I realize this will be a finding, I’m trying to make it so it doesn’t become a material weakness.
    Thanks…



  • We had a similar situation - small team on a system that is to be replaced soon so - due to size of team - need developers to be able to install. Our compensating control is as follows: detective control which captures when changes have been installed in the live system.
    In our case, the system sends an e-mail notification when changes are promoted to live (although a secure report tracking changes should also be fine). This notification is reviewed by an independent person (i.e. IT person without access to promote changes on this system). They confirm the changes are valid and sign-off.


Log in to reply