Cauldrons fill up when exposed to rain (albeit extremely slowly). Cauldrons also activate comparators based off of their water level. However, when they fill with rain, the comparator only updates when it fills to 100%; at 33% or 66% the comparator remains off unless it receives a manual update.
Code analysis by @unknown can be found in this comment.
Linked issues
Attachments
Comments
I think the problem is that the cauldrons aren't sending a block update. The comparator reads the correct level when it receives an update. If you want a temporary fix, you could just run a redstone clock or a piston constantly pulsing next to the comparator.

Still occurs in Snapshot 13w38c. As you can see, none of the comparators turned on in either case. As suggested, placing a block adjacent to them updates the comparators. Basically, the cauldron needs to do a block update when water is added via rain.

Still occurs in Snapshot 13w39a.

This report might be more effective if reworded. Something along the lines of "Rain filling Cauldrons not producing block update for Comparators."

Confirmed for
14w31a
Minecraft 1.8-pre 1
Confirmed for 1.9.3 pre-3

Please link to this comment in the description
The following is based on a decompiled version of Minecraft 1.9 using MCP 9.24 beta.
The reason for this is very likely that the method net.minecraft.block.BlockCauldron.fillWithRain(World, BlockPos)
is directly changing the block state instead of calling the method net.minecraft.block.BlockCauldron.setWaterLevel(World, BlockPos, IBlockState, int)
.
Confirmed for 1.9.4
Confirmed for 17w13b
Merged into MC-120986.
Confirmed.