TabletConnect: Benchmark Test Changes

As tablets evolved into more powerful computers, TabletConnect also needs to evolve its testing and benchmarking methods. When we first tested the Zenithink ZT-180/Enso Pad back in August 2010, only single core processors and not so powerful GPU processors were used. RAM memory was also limited to 256MB or at the most 512MB. Now the norm are dual core processors with GPU capable of 1080p HD output, 1GB of RAM, and more powerful GPUs. Because of this, we are making some changes to our benchmarking tests.

     First, we are replacing the outdated NeoCore app that was used to benchmark OpenGL-ES 1.1 graphics performance. This app, created by Adreno Graphics (part of Qualcomm), has not been updated since May 2010. Neocore will be replaced by NenaMark which benchmarks OpenGL-ES 2.0. This app is better to test the more higher graphics performance that the newer tablets have. Actually, Motorola's Xoom tablet does not even run Neocore.

      Second, we are somewhat replacing Softweg's Benchmark app which was designed for single core processors and 2D graphics. Again, this is another app that has not been updated by its developer in a long time (since December 2009). Since this app also test memory performance and file system performance, we will continue to use those scores. However, the CPU and Graphics tests will be replaced by SmartphoneBenchmarks.com's Smartbench 2011 which can test multi-core processors and 3D graphics performance. Smartbench provides two types of aggregate scores: 1) Productivity (CPU tests) ; and 2) Games (GPU tests). The CPU test is specifically a multi-core test.

      Third, we will eliminate the BenchmarkingPI test as it is redundant. In addition, the test is not a true benchmark of how fast the processor is.

      Fourth, we are introducing BrowerMark benchmark. This test is designed to measure browser performance in JavaScript and HTML rendering. The test is really to compare browser performance. We will use this to compare the stock Android browser when there has been changes to the browser.

      Lastly, we will now include results (in seconds) of our PDF rendering test. The purpose of the test is to calculate the number of seconds the tablet takes to render a complex PDF file. To standardize the test, we use the Repligo Reader app as the PDF reader. The test file was chosen because it contains a variety of pictures, fonts and graphic designs. Generally, the faster the tablet renders the PDF file, the better the processor. This is a “real-world' test. The number of seconds is measured from when the PDF file is clicked to open to when Repligo finishes reading the file.

      Benchmarks tell only part of the story. Different benchmarking apps may test different aspects of the CPU and GPU which would account for variance. One cannot look simply at one set of benchmark scores. One should view all in an aggregate.

      All current and future tests are performed three times but not consecutively. The results are then averaged to determine final score.

      These changes will be transitioned starting with the Motorola Xoom review. Both current and new tests will be performed on the Xoom. Going forward after that review, the changes will be fully in effect.

      (Note: These tests are only applicable to Android tablets. It becomes challenging to test tablets of different Operating Systems since there are no standard benchmarking tests that can be used across all platforms.)