11


I want to analyse the test coverage of our code , and therefore, installed the simplecov gem.

Our testing environement has 2 seperate project: REST API test (Java+Rest-Assured) and Web UI testing (Java-Selenium).
As you can see, we dont have unit testing inside the rails app, and we are testing using external projects.

I configured the simplecov gem as descriped in the tutorial and put this, in the rails script:

require 'simplecov'
SimpleCov.start 'rails'
puts "require simplecov"

When loading the app, I see the string I printed.

I ran both automation test projects, saw their printouts in the rails log, but I don't see any coverage of controllers/models, I see only small precentage of initializtion files of some gems and rails.

I searched the net, and tried putting the code phrase in boot.rb or even puma.rb and it returned the same results.



Any ideas?

EDIT

Nothing helped with all the comments, but I figured out something very interesting, in all cases, I only see the name of methods marked as tested, not the content (in controllers).
I tried to put the simplecov start phrase in both bin/rails, puma.rb, config.ru, environment.rb, all not given the desired results of code coverage.

YogevAbr
  • 289
  • 3
  • 18
  • 1
    There are a couple of things that might go wrong here. First, you should run puma single-threaded. SimpleCov's multi-threading support is limited. Second, have you tried using SimpleCov's at_exit hook? It will help you find out when and where SimpleCov stop "recording" code execution coverage: https://github.com/colszowka/simplecov#customizing-exit-behaviour . Third, Ruby's coverage might provide everything you need already, so you don't need SimpleCov in the first place: https://ruby-doc.org/stdlib-2.5.0/libdoc/coverage/rdoc/Coverage.html I hope you find that helpful. – wteuber Aug 22 '18 at 21:13
  • Our puma is running single threaded, also, what you shared about the ruby lib seems to be as per class, which can be very tough to control – YogevAbr Aug 23 '18 at 07:50
  • I dont understand " and we are testing using external projects. " – Simon Franzen Aug 26 '18 at 21:32
  • @SimonFranzen Tests are not rails unit tests, but using external projects like selenium and rest-assured. – YogevAbr Aug 27 '18 at 09:42
  • 1
    Then I am not sure, how SimpleCov can track those tests and the coverage. – Simon Franzen Aug 27 '18 at 10:32
  • @SimonFranzen According to their documentation it's possible, but it's not working for me. – YogevAbr Aug 28 '18 at 07:31
  • Well regarding your case that the methods names are hit (green) is typical behavior since the stack gets loaded and simplecov recognizes that the methods are there. But this means they are basically untested. Anyhow whats the "rails script" you are referring to where you put the simplecov starter? – Denny Mueller Aug 29 '18 at 08:24
  • You definitively have to put it into `bin/rails` after the shebang and before everything else. But also that is the important part: you have to run `bin/rails server` and not just `rails server`. At least this triggered changes in my coverage unfortunately I can't test much further since we have rspec going. – Denny Mueller Aug 29 '18 at 08:38
  • @DennyMueller I put it on bin/rails in the application directory. I also put some prints there in order to see it when running "rails s", and I do see the puts's. – YogevAbr Aug 29 '18 at 08:38
  • @YogevAbr see my second comment.. stupid 5 minute limit on comment edits – Denny Mueller Aug 29 '18 at 08:39
  • @DennyMueller Get the same, but now I've noticed that I get this message upon terminatin the rails app: Coverage report generated for MiniTest to /Users/... , what is MiniTest? this is not my tests framework, I don't use ROR test frameworks. – YogevAbr Aug 29 '18 at 08:47
  • @YogevAbr cant tell you why you get this. maybe you still have the gem in your gemfile. Who knows since we dont know much about your project. But at least you are at the point that simplecov gets correctly initialized. Now you jsut have to fiddle around with the configuration. – Denny Mueller Aug 29 '18 at 08:53
  • @DennyMueller I am actually at the same starting point, I had the same behavior before posting this question here, but thanks for your help! – YogevAbr Aug 29 '18 at 08:59

4 Answers4

2

I'm not sure simplecov can measure the whole rails app coverage... But I googled something that you can attach as a rack middleware:

https://github.com/danmayer/coverband

And it's output is compatible with simplecov. So it looks like it could be useful in your case.

As you mentioned in your question you're using puma. I suspect that, since it's multi-threaded, it spawns a few rails apps and their simplecov output overwrites each other's results. I'd try with the single threaded server like webrick - but this may make your tests slower (depending on how the tests are fired up really) or try the coverband gem.

Also - even if the server is single threaded - I'm not sure if each request would not overwrite simplecov's output.

Greg
  • 5,862
  • 1
  • 25
  • 52
0

Maybe you have to specify the paths

require 'simplecov'
SimpleCov.start do

  # add_filter '/admin/'

  add_group "Models", "app/models"
  add_group "Controllers", "app/controllers"
  add_group "Lib", "lib/"
  add_group "Helpers", "app/helpers"
end
Simon Franzen
  • 2,628
  • 25
  • 34
  • 1
    With these paths I get 0/0 (100%) coverage, which is probably wrong. – YogevAbr Aug 28 '18 at 07:57
  • @YogevAbr Have you tried using absolute paths, to make sure this isn't an issue about working directory? Like `add_group "Models", "/home/user/path/to/project/app/models"` – vmarquet Aug 28 '18 at 17:31
0

You need to to start SimpleCov before loading any of your files, so put these lines as early as possible in your ruby entrypoint:

require 'simplecov'
SimpleCov.start

You could see an example in one of my repos here: https://github.com/tareksamni/DockUp/blob/master/spec/spec_helper.rb

I do autoload my ruby code after starting SimpleCov. You need to the same as well:

require 'simplecov'
SimpleCov.start

require './autoload'
Tarek N. Elsamni
  • 1,718
  • 1
  • 12
  • 15
  • I don't have a spec helper, as I don't running any tests using the rails environment, which file is loaded first with rails 5? you know? – YogevAbr Aug 29 '18 at 07:52
  • Same results. I really start to think that I am wasting my time and it's a bug with simplecov. – YogevAbr Aug 29 '18 at 08:13
0

I had the same problem trying to get coverage for the e2e playwright tests. Meanwhile subprocess handling was added to SimpleCov in 2020, so I created unicorn conf for the CI

timeout          60
worker_processes 1
preload_app      true
listen           3000

app_path =  File.expand_path('../', __dir__)    
stderr_path "#{app_path}/log/unicorn.stderr.log"

if ENV['CI']
  require File.expand_path('../lib/tutuf/simplecov', __dir__)
  SimpleCov.command_name 'End-to-End Tests'
  SimpleCov.enable_for_subprocesses true
  SimpleCov.at_fork do |pid|
    # This needs a unique name so it won't be overwritten
    SimpleCov.command_name "#{SimpleCov.command_name} (subprocess: #{pid})"
    SimpleCov.formatter SimpleCov::Formatter::LcovFormatter
    SimpleCov.minimum_coverage 0
    SimpleCov.start
  end
end

Then started the unicorn server with

bundle exec unicorn_rails --config-file config/unicorn_test.rb --env test --daemonize

and ran the tests with

npx playwright test e2e --workers=1

The coverage is written to the disk on shutting down the server. To be able to post it to codacy, I added to the CI configuration

kill -QUIT `cat tmp/pids/unicorn.pid`
while [ -s tmp/pids/unicorn.pid ]; do
    sleep 2
done

to give some time to SimpleCov to flush the report to the disk before attempting to send it. Initially I started with wait time of 0.2s but it was insufficient and I increased it to 2s. Depending on the machine you run and the size of the coverage report you may need to increase it further.

Sava Lachezarov
  • 41
  • 6