I also think the difference could be visual/tactile. Something that is puffy could be inflated to a larger size, but I associate fluffiness with softness. For example, an airbag in a car is puffed up to a large size, but is not necessarily soft. It is inflated and the fabric is very tight around the enclosed air. A pillow, or a scarf, or a furry jacket may not be big in size but can be fluffy and soft. Some types of chips, are described as 'puffed' for example 'cheese puffs' -- but they are actually pretty crunchy when you bite them. Something puffed would seem to describe its size.
Hope this was not too confusing! Feel free to ask me follow up questions!