Automated Control Testing Frequency 1781



  • Well put. We actually just had an issue. One of our newer auditors inherited a revenue assessment from a departed employee. I was reviewing her testing with her and we sound a control that required recalculation of 50, yes 50 system generated bills. Clearly, someone wasn’t thinking when they chose this sample size for testing. Unfortunately, this assessment was not reviewed during rounds 1 or 2 of our controls optimizations, but certainly will be fixed for the upcoming quarter end.



  • Hello Folks,
    I would like to add further to my earlier thread. I would consider three as apt mainly to cover different models involving a transaction for e.g. payroll test may in come cases would require a parameter of FICA, Federal and State Taxes to be deducted. In other case, we may expect a Garnishment and 401-K deduction. Then we may expect a calculation for an hourly employee.
    This sounds sort of an overkill, but a sample size of 3 would live no stone unturned. Yes, you can live with one sample too. But, I am against it.
    Ofcourse, this determination of sample size is based on ITGC being deemed effective.



  • Does your recommended sample size relate to post baselining? Our advisors accepted a sample of 3 but required that sample be run against the system for each scenario. Therefore with our systems having a number of different flags involving different deductions, etc. for each entry that soon aggregated up to a very large sample



  • I would call this benchmarking. I have been given to understand that baselining is performed for a new client. Yes, this is post baselining exercise.



  • To follow along with what Chhaava said about preferring a sample larger than 1 let me give you an example of where this could be of a benefit.
    We implemented a new benefits system last year. When we tested the calculation of employee eligibility for participation in our 401K program, the individual performing the test didn’t realize that this being a system control he ‘should’ only test one sample. He pulled a sample of 20 or 30 employees. What he found, was that the system mis-calculated the eligible hours required for participation for all employees who accumulated eligible hours in a given month - the system actually gave double credit for one month (first month post-conversion) and allowed some employees into the plan a little bit early. Had he not tested multiple employees, he likely would not have found the error.
    This is not an endorsement for having a sample size greater than 1 for a system control, but rather a lesson that one may need to test for multiple factors when testing a system control to ensure that all parameters are covered. This cannot always be done with a sample of 1.



  • To follow along with what Chhaava said about preferring a sample larger than 1 let me give you an example of where this could be of a benefit.
    We implemented a new benefits system last year. When we tested the calculation of employee eligibility for participation in our 401K program, the individual performing the test didn’t realize that this being a system control he ‘should’ only test one sample. He pulled a sample of 20 or 30 employees. What he found, was that the system mis-calculated the eligible hours required for participation for all employees who accumulated eligible hours in a given month - the system actually gave double credit for one month (first month post-conversion) and allowed some employees into the plan a little bit early. Had he not tested multiple employees, he likely would not have found the error.
    This is not an endorsement for having a sample size greater than 1 for a system control, but rather a lesson that one may need to test for multiple factors when testing a system control to ensure that all parameters are covered. This cannot always be done with a sample of 1.



  • You bet Kymike.
    Yesterday only I made my team test application control on 401-K contribution and match. In this case we had to select two sample, one for part-time employee and one for a full time employee as they had different parameters.
    Therefore benchmarking confined to one sample may not be sufficient. We have to be open on the sample size for application control.



  • Now this is where you have really confused me. If you had baselined the system with a sample of say 3 per each parameter (required by my auditor) coupled with proper change management processes then there would be no risk of any systematic error and a sample of 1 would suffice.
    If there is still a genuine possibility of a systematic error then surely a sample of 3 would only suffice if there are no more than 3 parameters involved eg there should be a minimum of 1 sample for each scenario. Infact if that were the case would it not make sense to either re-baseline the system so that you know there is no systematic error or, if the baselining cannoy guarantee there is no risk of a material error, intoduce an overarching control that provided the necessary assurance?



  • With a test of 1 it is important to ensure that the test covers all relevant scenarios - this may require more than one transaction. With some automated controls it may not require any transaction i.e. you may be testing that the system does not allow you to enter transactions outside a range which you may test by attempting to enter a wrong transaction.



  • You baseline the system during the first year. Subsequent years you benchmark transactions once in a year provided material change management have not occurred multiple times during the year and the ITGC over ICFR applications are deemed effective.
    Instead of searching for an all inclusive sample, we are better off with another one or two sample to incorporate all parameters.
    I hope that this helps.



  • I follow what you are saying, it is a difference in terminology from my perspective. The nature of our systems dictate that each sample covers one parameter. Therefore when I refer to one sample it is one per parameter and no more but we will sample the system more than 3 times because we have more than three parameters.
    All that being said, with over 30 systems each with between 20 and 50 parameters, we have chosen in a number of cases to go with overarching reasonableness controls rather than system testing to satisfy ourselves that our figures are materially correct.



  • On a slightly different topic - with the benchmark / baselining. When you make an acquistiton and migrate the new subsidary onto your systems should you rebenchmark (as we were requested) for say system access rights or just benchmark the new users.
    In practice is just more of the same data but was wondering if there is an approved route to doing this? (nb the new users had ‘prior’ approval from the board into our system)



  • I understand your dilemma.
    You can endeavor to identify a particular sample that would meet all parameters breaking them into 3 samples. It is your call how to tackle the situation. You can work with the control owner to derive an efficient sampling strategy to get to the bottomline of application control assurance.
    Otherwise, I would recommend that you go in for one dummy transaction incorporating all parameters if the system can permit can permit backout of this dummy transaction.
    All the best.



  • Abu,
    I would recommend rebenchmark as this tantamounts to an in scope change management.



  • Hi Chavva
    thanks for the quick reply. - at least my time wasn’t wasted 🙂



  • My experience within Internal Audit and my time at PwC is as follows:
    sample size of 1. But, you must cover all of the possible parameters.
    I.e. Test to ensure that the system calculates the payroll correctly:
    Sample 1: enter valid numbers
    sample 2: Enter negative numbers
    sample 3: enter numbers way out of range. (like tax amount higher than salary)
    sample 4: etc



  • Sample sizes guidelines from IIA and external Audit have been:
    Nature of Control Frequency of Occurrence Min # of Items to Test
    Manual Many times per day (> 5,000 transactions/mo)
    60
    Manual Many times per day 40
    Manual Daily (365 per year) 20
    Manual Weekly (52 per year) 10
    Manual Monthly (12 per year) 3
    Manual Quarterly (4 per year) 2
    Manual Annually (Once per year) 1
    Programmed
    Test one application of each programmed control activity if supported by effective IT general controls Otherwise test similarly to a manual control (e.g., 60)
    IT General Controls
    Follow the guidance above for manual and programmed aspects of IT general controls


Log in to reply