Indeed, boutique is functioning as an adjective. A boutique solution is probably one that is specific, "tailor-made", "custom-made" or "tailored" to the particular problem. This is in contrast to a more "brute force" solution, which may be applicable to many situations but is not so well designed for this problem.
Another example of this is when comparing "bulge bracket" firms (eg JP Morgan, Goldman Sachs) or banks to "boutique firms" (maybe Evercore, Lazard). The idea is that boutique firms are usually much smaller than the bulge bracket firms, but might offer more personalised service or have a particular niche.
Similarly, boutiques in the context of fashion may be much smaller than department stores, but offer much more niche/unique items, with other services such as tailoring and personal assistants, rare items you can't find elsewhere, and so on.
Lastly, boutique in "boutique vodka" is similar, it is probably a smaller brand of vodka which may not be so widely consumed, but is likely quite highly prized, unique or expensive.