Using a furnace powered by coal within a smelting process of a whole stack (reproduced it with iron ore, gold ore and raw chicken [tested no other materials, so might be a general bug]) the first time collecting items seems to always award the correct amount of EXP. After that it seems to be quite random if EXP is awarded.
Very easy reproducible though. Works in 25-50% of the cases. I'm taking out items by shift-clicking and picking up (just click) alternating. Seems to work best with bigger stacks of items to smelt.
Edit:
Tested further using half a stack of cobblestone. Took the finished stones out by picking up (so just left-clicking) as soon as each stone was ready. First three stones did not reward EXP. Fourth did. I hope this is not intentional, but it seems new to me and I could not find any information online if furnaces should give random or fixed amounts of EXP.
There is a random factor in the amount of EXP given.
Specifically, if the total amount of EXP in the furnace is not a whole number when the items are taken out, then the fractional part of the amount of EXP in the furnace is the probability of adding 1 to the whole number part. (Non-ore items generally produce a fractional amount of EXP per item)