Alpo,
This is could be made to work, but there are hurdles to overcome, as you've found out and as I found when I was playing with it on a larger scale.
First: you need enough air to combust the fuel. (More than enough because unburned gas is a clear sign that you're making carbon monoxide. I understand wanting to play inside in the winter – I do, too – but the most enclosed I'd be for this is in my garage with the 8' x 10' door open during any test firings of the stove.) But back to the combustion process: you need more air. You need a way to induce more air flow. Mechanisms include: (1) the momentum imparted by the flow of butane through the jet (that's what moves atmospheric air into and through a BP stove, bunsen burner or your gas-fired kitchen stove). (2) chimney effect of a tall vertical pipe of hot gas that is lighter than the surrounding air. This is how a fireplace get's its air and why you have to "establish a draft" with a small fire or by burning some paper before starting a larger fire. (3) some forced ventilation with a fan.
Also, any of those mechanisms will more air in a larger pipe. Monmentum from the jet works fine when the "pipe" is the size of a room, tent, probably down to about 6", maybe 4". Going to sizes below that and I'd suggest you use chimney effect.
I'd suggest you start playing with the "Backcountry Boiler" or "Kelly Kettle" concept – a larger inner tube (2"-3") with a surrounding water jacket. But with butane instead of wood, you could put HX fins and vortex generators on the inside of the exhaust tube as is done in gas-fired water heaters. And I'd take a page from JetBoil and put a cosy around the outside of the water jacket. The taller you make such an arrangement, the more efficient it will be – more HX area, more energy extracted from the exhuast gases, and more chimney effect. Of course it is less compact and light, so it's a trade-off.
With the bigger set-ups I was playing with, (I was doing backpacking hot tubs for naked college students), the bounancy of the 4" tube gets to be a bother. It has to be air-filled and that displaces a large/heavy volume of water. I did a U-tube like you did, but I think a vertical arrangement is probably easier to make work, water-proof and keep submerged. When you get the basic, vertical chimney and burner working, you can search out a matching, tall, narrow pot.
I have seen some stainless-steel vacuum mugs (for your morning coffee on your way to work) that were tube-in-tube with a void between them. And a similar stainless steel water bottle. They have a vacuum jacket as is, but cut into it and you could have a water jacket. It would be a small water jacket but it would let you test the concept. The size is close to what you want: 2.5-3" inside, and MUCH better than what you've been using. $12 is cheap for water-tight, welded, tube-in-tube stainless steel. You wouldn't be out of the woods yet – there's thermal expansion to consider, so I'd suggest ALWAYS having water in the annular space because you don't want the inner tube at 500F while the outer one is at 70F. But with water in it, the inner tube would stay at 212F and it is designed for that.