Usefulness of Ackermann Function as Computer Benchmarker?

[Themohawkninja]

Hello,

In addition to my interests in all manner of computers, I also am quite interested in mathematics, and I recently discovered a highly recursive mathematical function called the Ackermann function.

As you can see in the article, there is a brief mention of this function being a useful tool for benchmarking computer performance, however when I looked up Ackermann-based benchmarks, only academic papers about the subject seemed to show up in Google. As a result, I ended up programming my own in C++. Once it was up-and-running, I tested in on both my computer (i7-2600k w/ GTX 580), and my roommate's (i7-4790k w/ ASUS Hero VII on-board graphics), and my roommate's computer scored about 25% higher. I'm not really sure if that accurately displays the difference in performance, because I can't seem to find consistent information on the GFLOP count of my roommate's processor.

Is this function an accurate way of measuring either CPU, GPU, or total computer performance?

In-case anyone wants to try this on their own computer, the .exe application and the source code in .h format are linked below. Ignore the broken "how it works" function. I have yet to get it to properly display the message that you can read in the source code, so if any programmers want to tell me how to fix it, that would be nice.

Application (EXE): https://www.dropbox.com/s/bh5c72j395xz1m7/A44.exe?dl=0

Source (H): https://www.dropbox.com/s/guavn0c364w97hz/Form1.h?dl=0

[K^2]

Lets start with the fact that A(4,4) is far outside the int range. And why, exactly, are you passing 64 bit integers to it as parameters, but return just a 32 bit int? But that's just minor gripes. Then there is your implementation with forms, which is going to prevent you from running a clean, well optimized code. If you want to write efficient code, write it in C. Finally, the only thing you are really "testing" with this implementation is the stack.

Edit: All of the references talk about Ackermann as the benchmark of optimizer, not benchmark of the computer.

Edited April 25, 2015 by K^2

[ZetaX]

Is this function an accurate way of measuring either CPU, GPU, or total computer performance?

Unless you do weird things, all you will test is your CPU's arithmetics and maybe memory and busses. Why would you expect a purely arithmetical thing to test a GPU¿

[K^2]

Modern GPUs are just general vector processors. Lots and lots of arithmetic operations are precisely how you test them. What GPUs are really bad at is any sort of branching, which would make Ackermann absolutely the worst thing to use a GPU for.

[Themohawkninja]

Unless you do weird things, all you will test is your CPU's arithmetics and maybe memory and busses. Why would you expect a purely arithmetical thing to test a GPU¿

I don't. I have a pretty limited knowledge of computer science at the moment, which is why I asked what this would ACTUALLY test.

- - - Updated - - -

Lets start with the fact that A(4,4) is far outside the int range. And why, exactly, are you passing 64 bit integers to it as parameters, but return just a 32 bit int? But that's just minor gripes. Then there is your implementation with forms, which is going to prevent you from running a clean, well optimized code. If you want to write efficient code, write it in C. Finally, the only thing you are really "testing" with this implementation is the stack.

Edit: All of the references talk about Ackermann as the benchmark of optimizer, not benchmark of the computer.

The program was meant to do the exact opposite of what it does (run for a given period of time, and count the number of times it called the function), but I was having issues with it either breaking a 64-bit integer, or calculating for such a short period of time that I couldn't get any real data from it. The code would look nicer if it wasn't for the fact that this isn't so much of a serious program that I would put in a portfolio for a job, so much so as it is a little experiment in mathematics and computer science. It gave me quite the eye-opener as to just how fast current processors are when I had to tell the computer to run 100 million iterations to get a decent number of milliseconds to go off of.

Also, what is the "stack" with reference to computer science here? Is that the order/speed in which instructions are carried out, or something completely different?

[andrewas]

The stack is an area of memory to which the current contents of the processor registers are pushed every time you do a function call.

[K^2]

You don't usually push all the registers. Just the "relevant" ones. (TBD by the compiler.) Unoptimized code will almost never push any working registers to the stack. What always goes onto the stack is the return address and almost always a base pointer. But stack is used for a lot more than that. Stack is used to pass variables to a function, it is used for nearly all local variables within a function, and it is used in intermediate operations of complicated algebraic expressions.

For example. Consider a very simple function.

int sum(int a, int 
{
	int c = a + b;
	return c;
}

If you compile it without optimization and call sum(3,4) from main, the stack will get the following workout.

1. Push 4 to the stack.
   
2. Push 3 to the stack.
   
3. Push return address in main to the stack. (Followed by jump to address of sum function in memory.)
   
4. Push current base pointer (belonging to main) to the stack.
   
5. Set base pointer to match stack pointer.
   
6. Decrement stack pointer by 4 to create space for c on the stack.
   
7. Use base pointer to read values of a and b, compute the sum, and use base pointer again to store it in c.
   
8. Copy answer from c to return register. (Usually eax)
   
9. Pop base pointer from stack. (Returns it to base of main.)
   
10. Pop return address from stack, and return to main function.

Now imagine all of that on top of your A(m,n) function. Basically, all it does is stack operations simply because of the way you've set up recursion. Processor does very little of actual algebra. It will spend, maybe, one cycle in a hundred doing math, and the rest of the time will be wasted on branching, function calls, stack operations, and all of the mess associated with it. (Branch predictions, cache predictions, etc.)

When you want to test computational capabilities of the CPU, you really want to avoid all of that nonsense. You want to serve mathematical operations in as predictable a manner as possible, making sure that cache remains consistent, and any branches you must have are very well predicted. Then all the CPU is doing is pulling values out of cache, does math, puts them back into cache, and it can do this very, very fast. That's where you can get tens of billions of operations per second.

GPU is kind of a different story. It's really bad at general computing. Especially branching. You generally want to pull as much of that out of GPU code as possible and have CPU look after it. If you can reduce your math problem to "Do this set of operations a million times," you can get a good GPU give you trillions of operations per second. That's why GPU is so awesome at image processing, numerical integration, and artificial neural networks. And, of course, actual rendering. All of these problems are reduced to, "Here is the math you have to do for each point, now do this lots."

Edit: If you can read assembly, the sum(a, compiles to the following on x86. (Again, unoptimized)

push ebp
mov ebp, esp
sub esp, 4
mov eax, [ebp+8]
add eax, [ebp+12]
mov [ebp-4], eax
mov eax, [ebp-4]
mov esp, ebp
pop ebp
ret

Edited April 26, 2015 by K^2

[Camacha]

Is this function an accurate way of measuring either CPU, GPU, or total computer performance?

No, for the simple reason that computer performance in dependant on so many factors, of which a major one is what you are actually trying to do. So even if you might be measuring a computer's capability to solve Ackermann Functions to perfection, any other task will be less accurately gauged. Many components are slightly or even very specialised towards their final purpose.