There's too much that I want to say to fit in comments. But since I'm not planning on doing your homework for you, let's see if I can explain the problems, then you can fix them.
So, starting at the top:
initialize:
xor rax, rax ; initializes rax
What does this "initialize" do to rax? Do you know? If this were class, I'd wait for you to answer, but I'm going to assume you already know that it sets rax to zero. With that in mind, what does this line do to rcx?
mov rcx, 0 ; initialize rcx
Obviously it sets it to zero. So... why are you using two different ways to set registers to zero? It's a bit confusing to read. Also, one is likely to be more efficient than the other, so why not just always use the one that's most efficient, right? Which one is most efficient? See this.
Moving on:
mov rbx, 0 ; clear register to store result
mov rdx, 1 ; initliaze rdx to base 10 (10^0)
We'll ignore the spelling mistake...
convertInputToDigits:
mov rcx, [count] ; initialize count
This doesn't "initialize" count. It reads the current value from the memory specified by count into rcx. Hopefully count is 8 bytes long. Since that's how big rcx is, that's how many bytes mov is going to read. You don't show count getting assigned a value, but I hope you are doing so. Otherwise rcx will just contain garbage.
mov rsi, buf ; point to start of buffer
add rsi, [count] ; move to end of user input, this part may be wrong
I'm not clear why you have these lines inside the loop. When you get to the bottom of the loop and jump back to convertInputToDigits, you don't want to reset rsi to buf and add 'count' again. You want to set rsi once, then walk backward count times.
Actually, you don't want/need to read count multiple times either.
mov rbx, [rsi-1] ; move the first rightmost number in rbx
Since rbx is 8 bytes long, this is going to read the 8 bytes at [rsi - 1]. As we've discussed, that's probably not what you intended.
sub rbx, '0' ; subtract hex value 30 for each character
; to get decimal value 0-9
mov rax, [rbx] ; move the result in rax
Are you trying to make a copy of rbx into rax? Cause that's not what this mov does. Instead it is going to try to read from the memory location specified by the contents in rbx. But what's stored in rbx isn't a memory location (it's supposed the be the number 1, right?). So I'd expect this will crash with an access violation.
If you really are trying to copy rbx into rax, that would just be mov rax, rbx.
Alternately, why are you reading the value into rbx and copying it? Why not just use rax in the first place? Looking at your comments, aren't you going to be using RBX for something else anyway?
mul rdx ; multiply result, power of 10(rax*rdx)
Ok, let's talk about what this does. In brief it's RDX:RAX = RAX ∗ RDX. While you only specify RDX, the rest is implied. Now, what does "RDX:RAX" mean? Well, if the values in RAX and RDX are really big, multiplying them together will give a result too big to fit in a single register. So mul uses 2 registers to hold the result. RDX will be the higher bits and RAX will hold the lower bits.
For 121, you are never going to have results big enough to fill more than one register, so RDX is always going to end up being zero. That's going to be a problem below.
add rbx, rax ; stores the total in rbx
Storing the total is a good thing, that's exactly what you are trying to do. But you are already using rbx for something else. So when you pass thru the loop a second time, rbx won't have the results from the first loop.
dec rsi ; decrement up until first character
cmp rsi, 0 ; check if we are at the end of input/converted all input to decimal
je convertToHex ; if so jump to conversion
Let's take a moment to discuss what cmp/je do. In brief, cmp does a comparison, then sets some flags that other instructions (like je) can use. je then looks at the flags set by cmp and jumps as appropriate. But it turns out that cmp isn't the only instruction that sets the flags. So does dec. So you can see if rsi hit zero by using the flags set by dec without having to do the extra cmp.
(BTW, if you ever DO need to check for zero without having done a dec or something, test rsi,rsi is slightly more efficient than cmp rsi,0).
But the more important problem here is that rsi isn't going to be zero. Using the example I gave in my comments:
If rsi is (say) 0x100, then the 3 bytes of "121" would be at 0x100, 0x101 and 0x102.
So, if we start at 0x103 (and typically the address will be much larger), then we decrement once for each of our three digits, we aren't looking for zero. So what could we decrement that currently has the value 3? <cough>rcx<cough>
mov ax, [rdx] ; move value of rdx to ax
Remember when I said that setting rdx to zero was going to be a problem? Well, here it is. If rdx is zero, then you are trying to read the memory location [0]. That's going to be an access violation. Since rdx is going to be zeroed each loop, perhaps you should pick a different register to store your power.
imul ax, 10 ; updates to the next power by 10*1 (10^1, 10^2)
This form of imul is going to effectively do ax = ax * 10. But is ax where you want to store the result?
jmp convertInputToDigits ; if not, loop again
So, that seems like a lot of problems. And, yeah, it kind of is. But you seem to correctly understand what you are trying to do, even if you are a bit fuzzy on exactly how to do it.
Hopefully this helps you understand what's going on a little better and points you in the right direction.
