Automated Control Testing Frequency 1781



  • Hi,
    I was wondering if anyone has any information on how many times per year an automated test must be tested. I was under the impression it was once a year, our auditors are stating twice per year. Any information would be greatly appreciated. Thanks



  • Unless major programming logic changes during the year, it is always advisable to benchmark application controls with 3 samples.
    Perhaps, in your case, there would have been a major change management.



  • What is the rationale for testing a system control using 3 samples? It sounds inefficient and redundant.
    As for number of times an application control must be tested yearly, it would be helpful if anyone has experience to share. How many times per year and what is the ideal timing to conduct the tests? It seems that, at year end might be best?



  • The rationale was conservatism which has been accepted and being followed by Big 4 auditors(except for Ernst and Young (the test one)).



  • Yes, I am actually looking for number of times tested per year…sorry, might have wrote it up incorrectly. We were holding off on automated testing for Q4 but were told we need to do the testing twice per year. Any comments on the number of times this must be performed per year would be great. Thank you



  • For systems with major changes and large shifts in users I recommend semi- annual. The main reason is to ‘catch’ any problems early in the year to avoid the crunch to get back in line at year end. If an application has had no major changes you should be able to argue once a year is sufficient. Your change management process should be strong and that would also strenghthen your arguement for less testing on the application.



  • We (my internal audit department) are strong proponents of one time a year as long as there have been no major updates in software. This has been approved by our externals too (PricewaterhouseCoopers)
    Hope this helps.



  • We use the same method as Jason (Deloitte)



  • Provided that you have effective IT General Controls…
    TEST OF ONE.



  • Gents,
    Thanks. Another SOX professional posed the question, but I found it interesting, and learned from the postings. I was not overly concerned about a recommendation suggesting that automated controls are tested one or multiple times during the year, but had serious reservation about a recommendation that suggested selecting more than ‘one’ in the audit sample…again, it seemed somewhat obvious that this was excessive and unnecessary.
    As a matter of information, I run a spell-check (not here though) only once before banging the send button–same logic and I think that double or triple checking a computer is a pointless exercise.
    Milan



  • Well put. We actually just had an issue. One of our newer auditors inherited a revenue assessment from a departed employee. I was reviewing her testing with her and we sound a control that required recalculation of 50, yes 50 system generated bills. Clearly, someone wasn’t thinking when they chose this sample size for testing. Unfortunately, this assessment was not reviewed during rounds 1 or 2 of our controls optimizations, but certainly will be fixed for the upcoming quarter end.



  • Hello Folks,
    I would like to add further to my earlier thread. I would consider three as apt mainly to cover different models involving a transaction for e.g. payroll test may in come cases would require a parameter of FICA, Federal and State Taxes to be deducted. In other case, we may expect a Garnishment and 401-K deduction. Then we may expect a calculation for an hourly employee.
    This sounds sort of an overkill, but a sample size of 3 would live no stone unturned. Yes, you can live with one sample too. But, I am against it.
    Ofcourse, this determination of sample size is based on ITGC being deemed effective.



  • Does your recommended sample size relate to post baselining? Our advisors accepted a sample of 3 but required that sample be run against the system for each scenario. Therefore with our systems having a number of different flags involving different deductions, etc. for each entry that soon aggregated up to a very large sample



  • I would call this benchmarking. I have been given to understand that baselining is performed for a new client. Yes, this is post baselining exercise.



  • To follow along with what Chhaava said about preferring a sample larger than 1 let me give you an example of where this could be of a benefit.
    We implemented a new benefits system last year. When we tested the calculation of employee eligibility for participation in our 401K program, the individual performing the test didn’t realize that this being a system control he ‘should’ only test one sample. He pulled a sample of 20 or 30 employees. What he found, was that the system mis-calculated the eligible hours required for participation for all employees who accumulated eligible hours in a given month - the system actually gave double credit for one month (first month post-conversion) and allowed some employees into the plan a little bit early. Had he not tested multiple employees, he likely would not have found the error.
    This is not an endorsement for having a sample size greater than 1 for a system control, but rather a lesson that one may need to test for multiple factors when testing a system control to ensure that all parameters are covered. This cannot always be done with a sample of 1.



  • To follow along with what Chhaava said about preferring a sample larger than 1 let me give you an example of where this could be of a benefit.
    We implemented a new benefits system last year. When we tested the calculation of employee eligibility for participation in our 401K program, the individual performing the test didn’t realize that this being a system control he ‘should’ only test one sample. He pulled a sample of 20 or 30 employees. What he found, was that the system mis-calculated the eligible hours required for participation for all employees who accumulated eligible hours in a given month - the system actually gave double credit for one month (first month post-conversion) and allowed some employees into the plan a little bit early. Had he not tested multiple employees, he likely would not have found the error.
    This is not an endorsement for having a sample size greater than 1 for a system control, but rather a lesson that one may need to test for multiple factors when testing a system control to ensure that all parameters are covered. This cannot always be done with a sample of 1.



  • You bet Kymike.
    Yesterday only I made my team test application control on 401-K contribution and match. In this case we had to select two sample, one for part-time employee and one for a full time employee as they had different parameters.
    Therefore benchmarking confined to one sample may not be sufficient. We have to be open on the sample size for application control.



  • Now this is where you have really confused me. If you had baselined the system with a sample of say 3 per each parameter (required by my auditor) coupled with proper change management processes then there would be no risk of any systematic error and a sample of 1 would suffice.
    If there is still a genuine possibility of a systematic error then surely a sample of 3 would only suffice if there are no more than 3 parameters involved eg there should be a minimum of 1 sample for each scenario. Infact if that were the case would it not make sense to either re-baseline the system so that you know there is no systematic error or, if the baselining cannoy guarantee there is no risk of a material error, intoduce an overarching control that provided the necessary assurance?



  • With a test of 1 it is important to ensure that the test covers all relevant scenarios - this may require more than one transaction. With some automated controls it may not require any transaction i.e. you may be testing that the system does not allow you to enter transactions outside a range which you may test by attempting to enter a wrong transaction.



  • You baseline the system during the first year. Subsequent years you benchmark transactions once in a year provided material change management have not occurred multiple times during the year and the ITGC over ICFR applications are deemed effective.
    Instead of searching for an all inclusive sample, we are better off with another one or two sample to incorporate all parameters.
    I hope that this helps.


Log in to reply