Yep - it would do the 100/100 first, giving 1, then add 20, giving 21, and then multiply by 30, giving 630, because of the order of mathematical operations.
If you want it to do the 100 + b.def first, you need to put that in brackets. In fact, since b.def is probably an integer, you likely will also need to convert to a float or (100 / (100 + b.def)) will end up as 0. And THEN you'll need to convert back to an integer at the end.
So try the following, in order, until it gives a value you're expecting:
a.atk * (100 / (100 + b.def))
if that gives you 0, try this:
a.atk * (100.0 / (100 + b.def))
if that gives you a number with a decimal part, try this:
(a.atk * (100.0 / (100 + b.def))).to_i