@bentzi
your problem seems like a bug in math library (libm.so). try upgrading glibc to newer version. it's also possible that your compiler optimization is breaking it. try compiling static binary with no optimizations.
Code:
LDFLAGS="-static" OPT="-g -O0"
@RedPenguin
does this happens also with rtmpsuck? also post the verbose log of successful and failed attempt.