I've been interested in hardware programming recently, but I have not started yet.
I did some searching working, and have a vague idea:
Arduino is a combination of both chip and breadboard.
AVR is a single chip, and need to buy a breadboard to get started.
Is that statement true or false?
AVR is just an integrated circuit microchip, made by Atmel. It looks something like this: 
Although they can be used by themselves, it takes a bit of hardware experience and some support components.
The Arduino is an AVR processor running special code that lets you use the Arduino environment to program and upload code easily. All you need is a USB cable to program and communicate with it. It looks something like this:
(source: mitchellpage.com.au)
A breadboard technically looks like this, and can be helpful in doing things with an Arduino. It is not necessarily needed for either solution but it is useful.
If your asking these sorts of questions, you should checkout the Arduino tutorials at adafruit. They're very basic and will teach you what you need to know to get started.
AVR is a processor architecture. It is a design for how the processor should work like x86 and ARM. Atmel sells hardware that has the AVR architecture in 8-,16-, and 32-bit architectures through products like ATTiny, ATMEGA, etc.
Arduino sells boards meant for hobbyist to put something together quickly and with little knowledge of the underlying hardware. These boards usually have ATMEGAs in them.
If you are new to programming at this "hardware" level, I would suggest buying an arduino board. You get a lot of bang for your buck since you don't have to buy breadboards, programmers, power supplies, etc. and it is much easier to work with most of the hardware is soldered onto the board.
AVR is a microcontroller of the ATMEL family, used in Arduino. ARM is a microprocessor. Most of the Arduino Boards come with AVR controllers. Recently Arduino Due has been launched which is based on ARM processor. Hope this is enough to clear-out things between AVR, ARM, Arduino(',')
What you've said is correct - the AVR is the family of the chip used on the Arduino. There's a whole bunch of 8bit AVRs with various flash/ram/peripheral/pin configurations, just as there are a bunch of different Arduino boards and clones.
However while software for the AVR is commonly developed in assembly, C or basic and loaded on to the chip using a hardware programmer, the Arduino includes a cross platform IDE (see the Wikipedia page) that is designed to make getting started much simpler. Software is developed in a C-like language (roughly based on Processing) and supports loading software on the device using just a USB cable.
What does it all boil down to? The Arduino will get you started quicker, and you'll see results faster (and they are becoming very popular in the maker/hobbyist community) - plus the Arduino Sheilds which plug in to provide additional functionality are a bonus. Getting a chip, breadboard and programmer will (arguably) give you slightly more power and flexibility - and etching your own circuit boards (optional) is a blast!
To use an analogy, Arduino is like a software framework, and the AVR MCU is like a programming language.
