2

I have asked this one earlier too but am not satisfied with the answer.

What I use:

  • Working on Django/python website.
  • Development done on python virtual envs locally.
  • using GIT as my SCM
  • have separate virtual servers deployed for Developer and Production branches for GIT
  • Using Jenkins CI for continuous Integration. Separate Virtual server deployed for Jenkins

Working:

  • I have Unit tests, smoke tests and Integration tests for the website. Jenkins has been setup so that whenever code is pushed from my local git branch to Developer and Production branch on git repo, a build is triggered in Jenkins.

Issue:

  • My tests are passing locally when I do a 'python manage.py test'
  • Random tests (mostly unit tests) FAIL in Jenkins when code is pushed to other branches (Developer and Production).
  • After a test failure, if I do do a build manually by pressing the 'Build Now' button in Jenkins, the tests usually pass and the build is successful.
  • Sometimes, when no changes are made to the code and code is still pushed to these branches, the tests are randomly failing in Jenkins.

Some common Errors:

  • AssertionError: 302 != 200
  • TypeError: 'NoneType' object is not subscriptable
  • IndexError: list index out of range
  • AssertionError: datetime.datetime(2012, 12, 5, 0, 0, 27, 218397) != datetime.datetime(2012, 12, 5, 0, 0, 27, 239884)
  • AssertionError: Response redirected to 'x' expected 'y'

Troubleshooting till date:

  • Ran all the tests locally on my machine and also on the virtual server. They are running fine.
  • Ran the individual failing tests locally and also on the virtual server. They are running fine.
  • Tried to recreate the failing conditions but as of now, the tests are passing.

The only problem I see is that whenever the code is pushed to the developer and production brnaches, the random test failure kicks in. Some tests fail repeatedly.

Can anyone tell me what more I can do to troubleshoot this problem. I tried googling the issue but in vain. I know xunitpatterns website has some good insights on the erratic tests behaviour but it is not helping since I tried most of the stuff there.

SaurabhM
  • 7,995
  • 1
  • 15
  • 20

2 Answers2

1

This is a really tough question to answer.

It is possible there are some common pitfalls django developers falls into, but those I do not know.

Outside of that, this is just normal debugging:

  1. Find a way to reproduce the failure. If you can make the test fail on your own laptop, great. If you cannot, you have to debug it on the machine where it fails.

  2. Get more information. Asserts can be made to print a custom message when they fail. Print values of relevant variables. Add debug printouts into your code and tests. See where things are not the way they are supposed to be. Google how to use the Python debugger.

Keep an open mind. The bug can be anywhere: in the hardware, the software environment, your code or in the test code. But unless you are god, Linus Torvalds or Brian Kernighan it is a safe first hypothesis the bug originates somewhere between your keyboard and back of your seat. (And all the three hackers above have made bad bugs too.)

sti
  • 11,047
  • 1
  • 27
  • 27
-1

For the issue - AssertionError: datetime.datetime(2012, 12, 5, 0, 0, 27, 218397) != datetime.datetime(2012, 12, 5, 0, 0, 27, 239884)

I have used the freezegun library to make date time more predictable. I am not sure why this issue comes but freezing date and time helps to some extent.

example:

from freezegun import freeze_time
import datetime
import unittest


@freeze_time("2012-01-14")

def test():

assert datetime.datetime.now() == datetime.datetime(2012, 1, 14)
flx
  • 1,560
  • 13
  • 22
Ashis Kar
  • 1
  • 1