Sample size - Isolation testing

[regisrosie]

What sample size do your external auditors require for isolation testing?

Specifically, let's say you do rollforward testing for a transactional control. You test 60 transactions in 3 rounds of testing (so 20 each round). In the first round 1 of the transactions fails. Do you test another 20 transactions to determine if the failure was isolated? Or do you test half of your original sample (10)? Or do you do something completely different.

Note I'm not asking about remediation testing. I'm asking about the determination of whether a failure was an isolated event, or if there was an actual exception for the current round of rollforward testing.

Thanks!

[gmerkl]

What do you mean with different rounds of testing? Do you mean different samples for transactions from different quarters or for different classes of transactions?

Are you doing management's assessment of ICFR or are you the registered public accountant that audits the effectiveness of ICFR?

I would say that the whole thing also depends on the size of the population from which the sample has been drawn that has been tested.

1 failing transaction out of a sample of 20 transactions means a failure rate of 5%. If your total population is much larger than 20, the total failure rate as a percentage of the population could be much smaller. The size of the sample has a direct influence on the likelihood that the sample is not representative for the whole population.

If you do management's testing and if management does not want an exception and is willing to spend your time (and thus resources), you can always increase your sample size and if no further exceptions are found, you can argue that it was an isolated exception.

[regisrosie]

Different rounds of testing: I mean different samples for transactions for the same control from different quarters.

I am an internal auditor doing management's assessment of ICFR.

We wish to increase our sample size and if there are no exceptions argue that the first exception was isolated. However, because our original sample size was not statistically based, we’re unsure how much to increase our sample size.

If our original sample size for Round 1 was 20, do we increase the sample by 20 more? Do we increase subsequent rounds’ sample sizes too? In essence, I’d like a feel for what others do. Our external auditors (the public accountants) will ultimately weigh in because they rely on our testing.

Or is the answer that we need to obtain a failure rate (1 error out of 40 sampled is 2.5%) that we would be comfortable calling isolated? Perhaps we determine a threshold? Is that what others do? A failure rate that exceeds a predetermined rate of lets say, 3%, is not isolated?

Thanks for your feedback.

[Denis]

Ultimately this will depend on your auditors, if you want them to rely on your work then you need to do it to their standards. The firm I used to work for would have extended the sample size by 20 for one exception and would have needed no exceptions in the extended sample to call it an isolated error.

A higher error rate and your getting into control deficiency and quantify error territory
_________________
"The art of life is to deal with problems as they arise, rather than destroy one's spirit by worrying about them too far in advance" - Cicero

[regisrosie]

[gmerkl]

There is no simple answer that can be based on the size of your sample only.

In addition, the fact that your sample was not statistically chosen is a problem in itself. As a consequence you cannot draw any conclusions from the sample about the total population. The sample may not be representative of the total population at all. In addition, your auditors may not be able to rely on your testing if the sample was judgementally chosen.

I would advise you to contact your auditors which sampling methodology and which sample sizes they would accept for given total populations for certain controls (i.e. a control that is executed multiple times per day results in a certain number of occurrences per year, the same yearly occurrences can be calculated for numbers of executions of controls that happen only weekly, monthly or quarterly). The degree of confidence that the auditors want and the size of the population drives the size of the sample that needs to be picked. The auditors should specify their maximum tolerable error rates. You can calculate the new increased sample size backwards using the actual number of errors in the old sample and the maximum tolerable error rate in order to arrive at the new sample size so that the number of errors that was already discovered as a percentage of the new increased sample size would be lower than the maximum tolerable rate of errors (provided that there are no new errors). The difference between the old and the new sample size is the additional number of items that you need to pick. If the auditors require the sample to be randomly chosen and if they require a greater sample size than your old sample, they will not rely on your testing anyhow, regardless of the number of exceptions that you identified.

It also depends on whether the auditor rely on your testing of controls only for the purpose of their audit of the effectiveness of internal control over financial reporting as of the end of the financial year or for the regular audit of the financial statements. In the former case the controls only need to be effective as of the end of the financial year. If they are ineffective during the year, but effective as of the end of the year (i.e. the sample is close to the end of the year), there is no problem. In the latter case, the controls need to be effective during the entire year or the auditors will have to use alternative audit techniques such as substantive testing instead of the testing of controls (or relying on your testing of controls).

[harrywaldron]

Hi - I agree with both SOX experts above ... You might want to search these forums using SEARCH button and keyword testing (although you're likely to get a ton of hits. Still, I've seen some good threads related to testing materially significant financial exposures on a daily through annual basis.

As SOX is somewhat a self regulatory program with a framework of sometimes "nebulus" guidelines, perhaps the best advice is to consult with your SOX external auditor (as they will help validate and sign-off on 404 compliancy).

Good luck and please continue to use the forums if you have any questions