The axiom "If you're going to roll your own security, you've already failed," applies here.
For your 5 character [0-9, A-Z, a-z] pin, you're generating less than 8.27 bits of entropy (64 310 = 2^n). [fixed]
It will take less than one day (a 1,000 guesses/sec, which is very slow) for an attacker to break your system. Is that acceptable? Maybe for trivial systems where bypassing security has very little impact.
Should I write a lockout mechanism to "ban" traffic from IPs with a large amount of failed attempts?
IPs can be spoofed.
Should I write an error checking system (similar to Luhn's algo in credit card numbers)?
That would actually decrease the number of bits in your entropy, making it easier to break into your system.
Should I make use of a captcha system?
If you feel you need the exercise. Captchas have been broken and are useless for anything other than as a speed bump.
Update
Unfortunately, there is no cut-and-dried solution for computer security, as it is still an immature (undermature?) field. Anyone who says, "Oh, do this-and-this and you'll be fine" is skipping one of the most fundamental issues around security.
Security is always a tradeoff
The ultimately secured system cannot be accessed. On the other end, the ultimately-accessible system has no barrier to entry. Obviously, both extremes are unacceptable.
Your job as a software developer is to find the sweet spot between the two. This will depend upon several factors:
- The technical expertise of your users
- The willingness of your users to put up with security
- The cost (time and money) to implement and maintain (i.e., a more sophisticated system will generate more support calls)
- The impact a breech would have on your users and company
- The likelihood of a breech: are you the US Department of Defense? Visa? You're probably under attack now. Bob's Bicycle Shop in Ojai, CA is on the other end of the spectrum.
From your question, I take it that you're effectively generating their "password" for them. What if you flipped it on its head? Make the pin their account and the first time they log into your system they have to create a password* that is then required from then on.
*Yes, if this is a bank, then this is not a good idea.