To convert a hexadecimal value to a number use the to_number function with the appropriate format string. For hexadecimal that is an 'x'.
select to_number('AA','xx') from dual
170
Now if your color picker returns a value such as FFEEDD for red, green and blue you will have to get the individual parts of that value and convert them to number. You can use substring for that.
substr('FFEEDD',1,2) -- red value
substr('FFEEDD',3,2) -- green value
substr('FFEEDD',5,2) -- blue value.
Convert these values to number.
select to_number(substr('FFEEDD',1,2),'xx') from dual
255
select to_number(substr('FFEEDD',3,2),'xx') from dual
238
select to_number(substr('FFEEDD',5,2),'xx') from dual
211
Put is all in a function and your done.