A foreign key can not be "conditional". The only "exception" to that rule are null values which can't reference another table by definition.
If I understand your question correctly, you are trying to implement a constraint that says "if refunded is true then refund_id must reference an existing transaction".
I don't think you need the refunded column at all. Because the flag refunded can be derived from the value of the refund_id using the expression: refund_id is not null.
If you do want a column like that, just create a view with that expression.
If you do insist on having the redundant refunded flag, you could setup a check constraint like this:
CREATE TABLE transactions(
id SERIAL PRIMARY KEY,
type TXN_TYPE NOT NULL,
amount BIGINT,
. . . .,
refunded boolean DEFAULT FALSE,
refund_id integer null REFERENCES transactions,
constraint check_refund
check ( (refunded and refund_id is not null or
(not refunded and refund_id is null) )
);