The size of the value returned from the query is the issue. HammerDB is using the TCL command "string is integer" to check the value returned from SQL Server. This command checks for a 32-bit integer and as your value exceeds this is returning an error. To correct this in the graphical transaction counter from the home directory edit the file in src\mssqlserver\mssqlsotc.tcl and find and change the line "string is integer" to "string is entier", it will look like this:
if { [ string is entier -strict $tc_trans ] } {
Similarly to correct this in the output of the OLTP/TPC-C test
edit the file in src\mssqlserver\mssqlsoltp.tcl
if { [ string is entier -strict $end_trans ] && [ string is entier -strict $start_trans ] } {
This will permit the larger values to be displayed.