Acceptable Failure Rates 859



  • Have you seen any criteria (informal or formal) on acceptable failure rates when testing controls for SOX?

    For example: if you pull a sample of 30, how many can fail and you still get a passing grade for that test?
    1 out of 30?
    2 out of 30? etc…

    also, if you fail, do you pull another sample and test again?
    if so, how many can fail in the additional sample?

    i know this is getting complex but it can make the difference between a minor exception and big failure. our external auditor will not provide guidance. they want us to develop our own standard and be prepared to defend it. we are willing to do that but want to ensure we are taking a solid approach since the external auditor always changes the rules when them come in later.

    what have you seen in terms of approach for acceptable failure rates?
    any general rules of thumb?
    any suggestions?



  • We (my company) do not tolerate failure, if one fails, then it’s enough, which D-and-T agrees to



  • For controls performed daily or multiple times per day, the guidelines we are using are: if one failure is found, then double the test size automatically. If no further failures are found then generally* the control would be passed. A second failure, either in the original or enlarged sample, and the control is deemed not to be working effectively.
    (* if the control is specifically designated to mitigate a significant fraud risk, then ‘one strike and you’re out’. One exception and it’s a fail.)
    In the case of controls performed less frequently (weekly or monthly, for example), then the same rules are applied, with the stricter criterion that even with only one failure, judgement should then be applied as regards the significance of the risk the control is attempting to mitigate.
    We also have D-and-T: I don’t know how our circumstances differ from IrquiM’s company, but it appears that we have a slightly less stringent approach.



  • It’s not D-and-Ts suggestion, but they agree to our approach.
    And it is probably run by different D-and-T teams too



  • For controls performed daily or multiple times per day, the guidelines we are using are: if one failure is found, then double the test size automatically. If no further failures are found then generally* the control would be passed. A second failure, either in the original or enlarged sample, and the control is deemed not to be working effectively.
    (* if the control is specifically designated to mitigate a significant fraud risk, then ‘one strike and you’re out’. One exception and it’s a fail.)
    In the case of controls performed less frequently (weekly or monthly, for example), then the same rules are applied, with the stricter criterion that even with only one failure, judgement should then be applied as regards the significance of the risk the control is attempting to mitigate.

    That’s the approach that I would follow as well. For some reason I seem to recall that Auditing Standard No.2 covers this, but I don’t have my copy to hand to check.



  • For controls performed daily or multiple times per day, the guidelines we are using are: if one failure is found, then double the test size automatically. If no further failures are found then generally* the control would be passed. A second failure, either in the original or enlarged sample, and the control is deemed not to be working effectively.
    (* if the control is specifically designated to mitigate a significant fraud risk, then ‘one strike and you’re out’. One exception and it’s a fail.)
    In the case of controls performed less frequently (weekly or monthly, for example), then the same rules are applied, with the stricter criterion that even with only one failure, judgement should then be applied as regards the significance of the risk the control is attempting to mitigate.

    That’s the approach that I would follow as well. For some reason I seem to recall that Auditing Standard No.2 covers this, but I don’t have my copy to hand to check.
    Well spotted. :oops: Page A-128 of the standard (example B-3) gives an idea of the PCAOB’s thinking on this.



  • Hi,
    I was hoping someone could post a link to the citation. I’ve been through auditing standard #2 on the PCAOB website but, I cannot locate the specific citation. Thanks in advance.



  • Hi,
    I was hoping someone could post a link to the citation. I’ve been through auditing standard #2 on the PCAOB website but, I cannot locate the specific citation. Thanks in advance.
    The example they give is as follows - and this is, to repeat, only an appendix to the standard so is presumably illustrative rather than definitive (if that makes sense… :roll: 😞 My underlining…
    Example B-3 Daily Manual Preventive Control
    The auditor determined that cash and accounts payable were significant accounts to the audit of the company’s internal control over financial reporting. Through discussions with company personnel, the auditor learned that company personnel make a cash disbursement only after they have matched the vendor invoice to the receiver and purchase order. To determine whether misstatements in cash (existence) and accounts payable (existence, valuation, and completeness) would be prevented on a timely basis, the auditor tested the control over making a cash disbursement only after matching the invoice with the receiver and purchase.
    Nature, Timing, and Extent of Procedures. On a haphazard basis, the auditor selected 25 disbursements from the cash disbursement registers from January through September. In this example, the auditor deemed a test of 25 cash disbursement transactions an appropriate sample size because the auditor was testing a manual control performed as part of the routine processing of cash disbursement transactions through the system. Furthermore, the auditor expected no errors based on the results of company-level tests performed earlier.
    a. After obtaining the related voucher package, the auditor examined the invoice to see if it included the signature or initials of the accounts payable clerk,evidencing the clerk’s performance of the matching control. However, a signature on a voucher package to indicate signor approval does not necessarily mean that the person carefully reviewed it before signing. The voucher package may have been signed based on only a cursory review, or without any review.
    b. The auditor decided that the quality of the evidence regarding the effective operation of the control evidenced by a signature or initials was not sufficiently persuasive to ensure that the control operated effectively during the test period. In order to obtain additional evidence, the auditor reperformed the matching control corresponding to the signature, which included examining the invoice to determine that (a) its items matched to the receiver and purchase order and (b) it was mathematically accurate.
    Because the auditor performed the tests of controls at an interim date, the auditor updated the testing through the end of the year (initial tests are through September to December) by asking the accounts payable clerk whether the control was still in place and operating effectively. The auditor confirmed that understanding by performing a walkthrough of one transaction in December. Based on the auditor’s procedures, the auditor concluded that the control over making a cash disbursement only after matching the invoice with the receiver and purchase was operating effectively as of year-end.



  • I personally agree with the testing approach described where the population is increased when an error is found. From my experience, our testing strategy for daily controls was to start with a sample of 25. When an error occurred, the error was evaluated to determine if the error was a significant error in the process or if it was an identifiable, isolated error. If it was the later, then an additional 15 items were selected to increase the overall sample size to 40. If an error occurred within the additional sample, the same procedure was performed again. If the error was isolated, then an additional 20 items were selected. If no error occured within the additional 20 then we were ok. If another error occurred, the control failed. %0AI know there are statistical probabilities associated with each of the population sizes (i.e. 25 = 90% assurance, 40 = 95% assurance, 60 = 98% assurance) but, I’m not sure about the specific %'s. If anyone could provide a link to support the statistical analysis of sampling, that would be very helpful. %0AWe are in a situation where the external auditor has stated that we should fail the control as soon as we identify one error, although we should test every sample. I’m not sure if I agree with this strategy and would like to be able to support the method described above.



  • I don’t have a link, but the IIA has an excellent resource that I have used to support our approach. ‘The IIA Research Foundation Handbook Series: Sampling: A Guide for Internal Auditors.’



  • Great. I just got my admission email from the IIA. I’ll have to look it up. Thanks.



  • Hello Everybody,
    I am new to this site and I am really happy I found it…
    This is what I have dealt with.
    I have a specific sample size I choose every month and if I have more than two exceptions then that section has failed. I know when we originally started last year if we had one exception then we would pull another sample but when our external auditors came at the beginning of the year they said that pulling another sample was not acceptable. That once you have two exceptions then you have to have a remediation. Has anybody else dealt with an outside auditor telling them the same?


Log in to reply