My android app is doing too much work on the main thread but I can't help it

Question

I'm writing a Android Whack-a-mole game. As we all know, the moles will pop up frequently on the UI and the timer text view is changed every second. So I used handlers for all of these works. I was told to use handlers instead of java.util.Timer in this question.

However, It didn't work the way I wanted it to. I mean it kind of worked but the timer text view's text changed irrationally. When I checked the logcat it said I'm doing too much work on the main thread.

Back then, when I was developing winforms programs, this problem never occurs. I was thinking about putting those handlers on another thread but as I know only the UI thread can access UI stuff, right?

Here is my countdown class:

package com.whackanandroid;

import android.os.Handler;
import android.widget.TextView;

public class CountDown {
    private Handler handler;
    private int timeLeft;
    private TextView textView;
    private boolean paused;
    private CountDownListener listener;

    private Runnable countDownTask = new Runnable () {
        @Override
        public void run() {
            if (!paused) {
                timeLeft--;
                displayTime ();
                handler.postDelayed (this, 100);
                if (timeLeft == 0) {
                    pauseCountDown ();
                    if (listener != null)
                        listener.onCountDownFinished ();
                }
            }
        }
    };

    private void displayTime () {
        textView.setText (Integer.toString (timeLeft / 10));
    }

    public void pauseCountDown () {
        paused = true;
    }

    public void startCountDown () {
        paused = false;
        handler.post (countDownTask);
    }

    public void setCountDownListener (CountDownListener listener) {
        this.listener = listener;
    }

    public CountDown (TextView tv) {
        textView = tv;
        timeLeft = 600;
        handler = new Handler ();
    }
}

And here is my phone class (the "mole")

package com.whackanandroid;

import android.os.Handler;
import android.view.View;
import android.widget.ImageView;

import java.util.Random;

public class Phone {
    private ImageView image;
    private Handler handler;
    private Random random;
    private State state;
    private boolean paused;

    private Runnable appearTask = new Runnable () {
        @Override
        public void run() {
            if (!paused) {
                if (random.nextInt (3) < 2) {
                    setState (State.ANDROID);
                    Game.getInstance ().addScore (10);
                } else {
                    setState (State.APPLE);
                    Game.getInstance ().addScore (-5);
                }
                handler.postDelayed (disapperTask, random.nextInt (1000) + 700);
            }
        }
    };

    private Runnable disapperTask = new Runnable () {
        @Override
        public void run() {
            if (!paused) {
                setState (State.NONE);
                handler.postDelayed (appearTask, random.nextInt (2000) + 1000);
            }
        }
    };


    private View.OnClickListener imageOnClick = new View.OnClickListener () {
        @Override
        public void onClick(View v) {
            //TODO add OnClickListener
        }
    };

    public void setState (State value) {
        state = value;
        switch (state) {
            case NONE:
                image.setImageResource (R.drawable.phone);
                break;
            case ANDROID:
                image.setImageResource (R.drawable.androidphone);
                break;
            case APPLE:
                image.setImageResource (R.drawable.applephone);
                break;
        }
    }

    public State getState () {
        return state;
    }

    public void stopTimers () {
        paused = true;
    }

    public void startTimers () {
        paused = false;
        if (getState () == State.NONE) {
            handler.postDelayed (appearTask, random.nextInt (2000) + 1000);
        } else {
            handler.postDelayed (disapperTask, random.nextInt (1000) + 700);
        }
    }

    public Phone (ImageView view, Random r) {
        view.setOnClickListener (imageOnClick);
        image = view;
        random = r;
        handler = new Handler ();
        image.setImageResource (R.drawable.phone);
    }
}

I think my code just looks horrible. Hope you don't mind. If you need any more code, please tell me at once.