Update: Not a Mod. But I don't know what the culprit is except it has to do with YaBB fopen/fclose routines.

The fix: in Admin/Backup.pm find

    # open in binmode
     fopen( READ, $filename )
       || fatal_error( q{}, "$backup_txt{46} $filename", 1 );
     binmode READ;

     # stream it out
     binmode STDOUT;
     while (<READ>) { print; }
     fclose(READ); 

replace with

    # open in binmode
     open( READ, $filename )
       || fatal_error( q{}, "$backup_txt{46} $filename", 1 );
     binmode READ;

     # stream it out
     binmode STDOUT;
     while (<READ>) { print; }
     close(READ); 

This uses Perl's native file open/close routines and bypasses any file locking or other things happening in YaBB's fopen/fclose routines.

Note: As far as we know this is a very rare occurance - at least it hasn't been reported as far as I know.
The key symptom is, if you're not using MSIE (which throws an error to the screen), when you look at the size of your downloaded compressed backup, the size is 1kb or less when the original is much much larger.