General answer is, that you must learn to use debugger, you must be capable to single-step over instructions, and check any register value, and any memory value, at any time, so you can validate/reason about your every assumption/expectation. From there it will be sort of simple to find problems like this, in debugger it will be quite obvious what is wrong with your code and why AAA is outputted.
Programming in assembly without debugger is like assembling robot blindfolded. You can do it, but it takes lot more effort (about 100x times more if you are talented, 1000+x or making you completely incapable to finish your task if you are not talented - and I'm not exaggerating this one, the 100x is quite conservative estimate from my experience, as I had to fix my code without debugger for years (and without computer, just with paper and pen), so I can compare).
Some notes... your functions are barely "readhex" and "printhex", more like "readstring_only0to9AtoF_valid" "writestring_with_hex_prefix". "read/write hex" sounds a bit more like there will be also conversion to/from binary value.
lea di,[string] should be lea edi,[string], makes me a bit wonder, why the code works for you without that, you probably did set edi to some other valid address in .data before, so the upper 32 bits are already OK, or the assembler is silently fixing it into edi, or the address is value < 65536 (most unlikely), check disassembler in debugger if you got really lea di, and what value is in edi before the lea. So you know exact reason, why it works by accident for you, even with the bug (this bug may be actually hard to spot in debugger - as long as it works by accident - if you are not reasoning about every instruction deeply).
printhex: problem:
mov ecx,ebx
lodsb
Loads first character of input, and adjusts esi - two problems:
- this happens outside of loop, so you will output the same char inside loop (being lucky enough, that
writechar does not modify al).
- this is done before
cld, so would you have DF=1, it would decrease esi, then inside the loop the cld is "late" switch to forward direction.
Actually your whole code looks to be designed as if it expects DF=0 all the time, so put single cld somewhere near the start of your code, and forget about it (unless you will add some function with std, then it should restore cld before exit).
.print:
call writechar ;function to print a char from al
cld
loop .print
Why loop is slow - rather use dec ecx jnz .print. But then you can use the ebx as counter, so you can remove also mov ecx,ebx and keep ecx intact, using dec ebx + jnz to loop.
And the cld is quite useless inside the loop. If you need it, it should be used before first lodsb.
Another problem I just guess because you didn't show your data definitions (which is sort of very bad in the question about assembly, data - values and structure - are often more important than code, so leaving them out is wrong, don't do that again).
mov eax, prefix ;prefix is a string containing "0X"
call writestr ;function to print a string
The prefix string is probably missing nul terminator, and the string buffer is defined right after it. So this call writestr will output whole 0XAB1 memory, before accidentally finding some zero to stop.
The remaining AAA output is from your character loop. Again a bug, which would be obvious from debugger, as after the call writestr to display prefix you would have on screen already whole number, not just expected "0X".
The readhex: has invalid handling of invalid characters, it will reset ebx counter, but not the input buffer pointer, so input like ABx1 will set input buffer to AB1, but return only 1 in ebx.
I would probably re-arrange the whole check a bit:
readhex:
mov ebx,edi ; remember start of input buffer
.nextchar:
call readchar ;function to read a char and store it in al
;jumps to the end if Enter
cmp al,13
je .end
call writechar ;function to print a char from al (echo input)
; check if input character is valid 0-9A-F
cmp al,'F' ; 'F' is maximal valid value
ja .errorchar
cmp al,'0' ; '0' is minimal valid value
jb .errorchar
; here '0' <= al <= 'F' - now just verify the range between
cmp al,'9'
jbe .validchar ; AL is '0'..'9', accept it
cmp al,'A'
jae .validchar ; AL is 'A'..'F', accept it
.errorchar:
call writeln ;function for linebreak
mov eax, errormessage
call writestr ;function to print a string on the screen
call writeln
mov edi,ebx ; reset input buffer
jmp .nextchar
.validchar:
;stores the valid character
stosb
jmp .nextchar
.end:
; calculate input length and return it in EBX
sub edi,ebx
mov ebx,edi
ret
