Sample Sizes and testing -update 1780

  • Just to let you know:
    I was speaking with a partner from the New York Offices of KPMG and was told that the PCAOB reviewers are constantly indicating that sample sizes and level of testing are not too large and that they are in many cases too small.
    This conflicts with the statement that emphasis should be with the ‘tone at the Top’, therefore reducing detailed testing.
    it was suggested that the reason may be due to the fact that the PCAOB file reviewers gain commission payments based on the number of deficiencies identfied.

  • Hi,
    Thank you for sharing your experience and providing feedback. Your observation increases the need for companies to inasmuch as possible, to convert manual controls to automated controls.
    It is my understanding that automated controls require testing only once, and if effective change management practices are in place, the result would likely lead to significantly reduced ongoing testing of the ICFR and/or disagreement between the client/auditor about appropriate sample sizes.

  • I agree.
    Automated controls allow for much more consistency and reliability.

  • I agree that automated controls do provide consistency and reliability. What confuses me though is that for us to introduce such controls we have to fully baseline our systems which has proven to be costly both directly and indirectly eg one system cost was equivalent to 50% of our audit fee (ignoring staff time). Our auditors then informed us that we had to repeat the same exercise every 3 to 4 years (irrespective of change management controls, etc) which is leading me to question whether there really is a cost benefit at all.

  • Hi,
    Yes, I agree that baselining systems is costly and the ROI is questionable when it must be repeated every 3 to 4 yrs. However, other benefits are derived by converting manual controls to system controls and baselining the application controls.
    Some potential benefits:

    • Significant efficiencies are gained after baselining the application(s) during the second or future iterations. For example, many companies are now reporting that Yr 2 and Yr 3 SOX compliance costs are much lower than incurred during the first year. These cost reductions were likely achieved because companies simply updated the SOX business process documentation and related test plans in subsequent years. Additionally. the ‘learning curve’ is scaled by those companies in Yr 2 or Yr 3.
    • Potential reductions in the resources necessary to baseline a system can also be achieved by focusing on the ICFR and other CobiT controls embedded within the system. Similar to the approach used to identify the relevant ICFR for control design and operation evaluation, this approach can also be used to reduce the need to test all application controls in the system(s) during baselining as opposed to the relevant controls for SOX compliance purposes.
    • Generally. much greater reliance can be placed on the efficacy of automated controls versus manual controls. For this reason, control risk is significantly reduced, audit risk is reduced too, and as a result, the incremental cost of baselining is ‘offset’ in non-financial terms from the risk reduction and likelihood of a material control weakness arising from an ineffective control.
    • Recurring personnel and SOX assistance resources necessary to perform tests of manual controls is reduced and if controls testing is performed by company management, shifting this process via a baselining approach reduces the workload on management thereby allowing them to focus on running the business.
    • Finally, it is possible to transition automated controls to continuous controls monitoring and thus, point in time assessment is replaced with assessing controls in a real time environment. For this reason, other aspects of SOX, such as identification and reporting of real time disclosure of material events, is more easily accomplished.

  • Totally agree with this.
    We have actively challenged all of our key control selection in the last 12 months, encouraging process owners to focus on automated and monthly controls rather than daily/transactional controls.
    In many cases we have actually designed new month and quarter-end activities to facilitate this.
    It has saved us a lot of effort.

  • From what we have been told by our Ext Auditors the testing sample size guidance is:
    Nature of Control Frequency of Occurrence Min #of Items to Test
    Manual Many times per day (> 5,000 transactions/mo)
    Manual Many times per day 40
    Manual Daily (365 per year) 20
    Manual Weekly (52 per year) 10
    Manual Monthly (12 per year) 3
    Manual Quarterly (4 per year) 2
    Manual Annually (Once per year) 1
    Test one application of each programmed control activity if supported by effective IT general controls otherwise test similarly to a manual control (e.g., 60)
    IT General Controls
    Follow the guidance above for manual and programmed aspects of IT general controls

Log in to reply