Every counterparty action is a bitcoin transaction that requires a fee to get the transaction into the bitcoin network.
Your transactions were written to the bitcoin blockchain, so there is no refunds.
It sounds like your probably trying to use BTC on the dex without fully understanding that BTC orders on the DEX work a bit differently and require an additional BTCPay transaction.