Sample Sizes and _and_quot;cycles_and_quot; 687



  • The daily sample size seems excessive, but depending on the number of key controls you are testing (you are only testing ‘key’ controls, right?) this may or may not add a lot to your test load.
    Since this is in the IT forum, I am assuming that you are referring primarily to manual controls within the IT function (adding new user security, etc.). The idea behind sampling is to test a sample size that will give you statistically accurate depiction of the population as a whole. You may need to revisit the items that you are testing to see what your annual volume of control instances is and refine your test size to reflect an appropriate sample size. The idea is to test a representative sample of the controls, not test all of them. Maybe your auditor needs a refresher in statistical sampling :roll:



  • agree with you.
    yes we are testing only key general computing controls. there are 59 key controls identified. i think that is too high but that is what we have.
    regardless of the total population we have in IT, the external auditor still wants the large sample sizes. This is a significant increase in the effort required to test the controls.
    this is medium sized company. it does seem like overkill.



  • Hello,%0AI am an IT Auditor with a big four firm.%0AOur audit size selection is:%0ADaily or more often = 365+ we sample 25%0AWeekly or more often = 52 364 = 15%0AMonthly will sample 5%0AQuarterly we will sample 2%0AAnnually we will sample 1.%0AI should also state that we try to get to a 95% rate.%0AThis means that if we would sample a control and pull 25 and find one that is an exception we would still pass the control. If we found 2 out of the 25 as exceptions we would pull another 25 and see if we got a clean 25 selection in the next sampling. If we didn’t get anymore exceptions we would pass the control.%0ATherefore 2 exceptions out of 50 would be allowable, but three exceptions would fail the control.%0AI would also agree that many IT Auditors are not aware to the realities of IT. I know that exceptions will happen or you may have an SDLC that you only use once or twice a year for in scope systems, so I would then say OK, prove to me you only used it twice. If your proof is reasonable I say OK that falls within 1 4 times a year so I will test two.%0AI think this makes practical sense.



  • SOX_Monster, I am guessing you are working for E-and-Y. They are the only Big 4 I have worked with so far (others are KPMG and PWC) that cap the sample size at 25.
    These are the guidelines PWC and KPMG use:
    PWC:
    annual - 1
    quarterly - 2
    monthly - 2 to 5
    weekly - 5 to 15
    daily - 20 to 40
    multiple times a day - 25 to 60
    KPMG:
    annual - 1
    quarterly - 2 to 3
    monthly - 2 to 4
    weekly - 5 to 10
    daily - 15 to 30
    multiple times a day - 30 to 60
    They break it down by risk. For instance, a control reoccuring several times a day that is high risk would be a sample size of 60. Low risk, a sample size of 30.
    After a while, this becomes absurd. Pulling 60 screenshots becomes a tedious exercise in excessiveness.



  • :lol:
    I agree, Pulling 60 samples and getting screen shots is tedious.
    I have never had to pull that many so I guess I am lucky.
    I also didn’t say we capped the sample at 25 as if we find two exceptions we pull another 25.
    I usually get someone else to pull my screen shots for me and do the documentation on them as well :lol:
    In the two SOX jobs I was on last year I was the manager, so if I knew there were screen shots from you know where coming up, I got the first years to do that test, and then I just reviewed their work.
    Of course my work was then reviewed by the senior manager, and his work was reviewed by a partner, and then the national risk partner…



  • why do you need to pull screenshots? that’s more than is required to evidence testing.



  • SOX_Monster, I am guessing you are working for E-and-Y. They are the only Big 4 I have worked with so far (others are KPMG and PWC) that cap the sample size at 25.
    My company use D-and-T which has also given us a cap at sample size 25



  • As far as I am concerned you don’t need to pull 60 screen shots.
    But you do need evidence.
    Off the top of me head I cannot think of anytime that we actually pulled 25 screen shots for one test.
    I usually tell my people to document the testing steps, pick you selection, get a screen shot or two of a couple of the tests and then document that you observed the rest of the selections.
    We take an attitude that if it can be reproduced at a later date, i.e. a screen shot of an access log, then one screen shot if any, is all that is required.
    Screen shots of 25 different computers showing that they had a viable anti-virus that was up to date at the time of the test may be required though. (I actually conducted that test myself and I just attested to the up to date AV)



  • One thing that should be considered as well is if it is an automated control, or a manual control



  • for a daily process, we were given a sample size of 50-60. and yes they do expect screen shots or some kind of evidence for all of them. an important clarification we just got is that the sample size is on an annual basis. this is important for us because we are doing sox testing now and we will do it again in 4th quarter. this means we would pull 25-30 now and then another 25-30 for the later round of testing.



  • Has anyone heard of a ‘rule of thumb’ for samples sizes that if you know the total population then you pull 30% for your sample (even when the external auditor or test script asks for a larger amount)?



  • Hi ugogirl,%0AThis sounds like it would lead you to performing a lot more testing than is necessary for the more frequent controls. For example, if a control were exercised on a daily basis, the population would be 365 over a year. 365 x 30% = ~110 samples. Not to mention some controls may be execised 3 times per day or more… 8O %0AOn the other hand, for a monthly control, this would lead you to a sample size of 4, which is in-line with the big-4’s. I think applying the 30% rule to anything beyond a weekly control frequency would create unneccessary burden. Maybe 30% for controls exercised weekly or less… 20% for annual controls… 15% for controls exercised more than twice a day…? Someone better with statistics could probably prepare the %0A’assurance’ curve for this one… 😮 %0AGood point about the control testing being performed on an annual basis - I know that some companies perofrming quarterly testing have not taken this into consideration and have ended up doing a lot more testing than would otherwise be necessary.%0ACheers,%0Alordkukuface



  • for a daily process, we were given a sample size of 50-60. and yes they do expect screen shots or some kind of evidence for all of them. an important clarification we just got is that the sample size is on an annual basis. this is important for us because we are doing sox testing now and we will do it again in 4th quarter. this means we would pull 25-30 now and then another 25-30 for the later round of testing.
    If the evidence of the control is a screenshot then this suggests automated control to me. If you are looking at automated controls then you can go down a GCC test of one route.
    Can’t imagine anything more pointless or soul destroying than pulling 50 screenshots for one control 8O



  • actually the screen shots are to show the approvals from user managment and IT management (external auditor wants to see: approval to start the project, approval of test results, and approval to migrate to production). these approvals are done via a software product that does help desk tickets, workflow, and change management. the only way to get the evidence is screen prints unfortunately.



  • You don’t need to keep any document as testing evidence that you can easily reproduce. Just ensure that your testing write-up covers what you tested and includes enough information to reproduce that testing. Usually, a test matrix with the attributes tested, the results of the tests and a conclusion as to the effectiveness of the controls is adequate.
    If you think that it is easier to keep the documents than reproduce later, that is a decision that you will have to make.



  • the external auditor has stated they want evidence in hardcopy stored in binders as part of the working papers. otherwise, we would take the easy route.



  • Who is your external auditing company?



  • It is certainly easier for the auditor to access if he doesn’t have to wait for you to recreate the screenshot. This may help to reduce auditor time and fees (though I can’t imagine that it will save them that much time)



  • HP,
    It would frowned upon here if I told you the name of the external auditing firm. However, I can say it is not one of the big 4.
    Thanks,



  • the external auditor has stated they want evidence in hardcopy stored in binders as part of the working papers. otherwise, we would take the easy route.
    Tell them to off :evil:
    If the evidence is held electronically then they need to review it electronically. SOX does not require you to do unnecessary work because your auditors are incompetent.


Log in to reply