In our current deploy setup, there exists more than 200,000 lines of code for some of our apps. Naturally, this means there are a LOT of unit tests paired with this code which need to be run, and that code coverage reports take a long time. Running the unit tests by themselves (nearly 2600 tests) took around 30 minutes to complete. However, adding in the code coverage to that run bumped the time up dramatically, to nearly 3 hours:
./vendor/bin/phpunit --coverage-clover ./tests/coverage/clover.xml
...
...
... (a lot of unit tests later)
Time: 2.88 hours, Memory: 282.50MB
OK (2581 tests, 5793 assertions)
Generating code coverage report in Clover XML format … done
Generating code coverage report in HTML format … done
The dilemma
In the existing setup, our deployment service received a webhook from our source code management software every time code was merged to the develop branch. The deployment service then pushed the code change to the server, ran our ansible deployment scripts, and then ran unit tests on the actual develop server environment. This was not ideal, for a few reasons:
- Bad code (malicious or vulnerable code, code that breaks functionality, or code that just doesn’t work) could be pushed to the server without testing happening first.
- Things could be left in a broken state if the deployment were to fail its unit tests, with no real accountability to fix the issue.
- The unit tests take so long it was causing the deployment service to reach its 40 minute timeout just on the unit tests, not even including the code coverage.
In a more ideal world, the deployment to the develop server environment should be gated by the unit tests (and security scanning as well) so that code is only deployed when tests are successful. And, the most ideal way to do this would be with an automated CI/CD pipeline.
We already had some regression testing setup in Jenkins, so creating a pipeline was certainly an option. The dilemma, however, was how to generate code coverage feedback in a reasonable amount of time, without waiting 3 hours for said feedback. Enter phpdbg.
The solution
phpdbg is an alternative to xdebug, and is an interactive PHP debugger tool. Unfortunately the documentation has very little information on usage or installation, but does mention that PHP 5.6 and higher come with phpdbg included.
That information, plus a few promising blog posts (including one from Sebastian Bergmann of phpunit himself and one from remi repo’s blog) gave us hope for a faster solution:
- http://kizu514.com/blog/phpdbg-is-much-faster-than-xdebug-for-code-coverage/
- https://hackernoon.com/generating-code-coverage-with-phpunite-and-phpdbg-4d20347ffb45
- https://medium.com/@nicocabot/speed-up-phpunit-code-coverage-analysis-4e35345b3dad
- https://blog.remirepo.net/post/2015/11/09/PHPUnit-code-coverage-benchmark
- https://thephp.cc/news/2015/08/phpunit-4-8-code-coverage-support
If this tool worked as promised, it could save a massive amount of processing time for very similar code coverage calculation results, and a little bit more Memory. Relatively small trade-offs for some big benefits, if you ask me.
Making the solution work
As it turns out, the silver bullet was more like a “bang your head on your desk until it works” kind of solution. What I read was promising, but I kept running into issues in execution.
- First, since our Jenkins instance had PHP 7.2 installed, it sounded like phpdbg should work right out of the box since it’s included in PHP from version 5.6+, right? Unfortunately, phpdbg wasn’t an available bin to be used, and wasn’t one of the packages installed with yum on our CentOS 7 servers.
- This github (now archived) from user krakjoe indicated if I just installed PHP from source using this repo it would work, but this too failed (and caused all other PHP functions to stop working).
- Eventually I stumbled upon these remi rpms that actually include phpdbg. The fun didn’t stop there, though…
- Firstly, installing the yum package worked well enough, but it took me a minute to realize that the bin is actually under “php72-phpdbg” and not just “phpdbg”. No big deal, so far…
- Now I actually had the php72-phpdbg command working and could enter the command line, but when I wrapped the phpunit commands with it, I was getting errors about other php packages (intl, pecl-zip, etc) not being installed. It turns out the php72-phpdbg package was from the “remi-safe” repo, which didn’t recognize the other php packages (which had been installed with the remi-php72 repo). To fix this, I had to install all the remi-php72 packages with the remi-safe repo instead.
At the end of the day when the dust settled, we got the results we were hoping for:
php72-phpdbg -qrr ./vendor/bin/phpunit --coverage-clover ./tests/coverage/clover.xml
...
...
... (a lot of unit tests later)
Time: 36.37 minutes, Memory: 474.50MB
OK (2581 tests, 5793 assertions)
Generating code coverage report in Clover XML format … done
Generating code coverage report in HTML format … done
Our coverage generator showed results were about half a percent difference lower than with phpunit alone (using Xdebug). Some users have reported coverage differences more than this, or are more concerned about the differences. For us, the difference is not in our favor (lower than original results), so we are less concerned. The benefits far outweigh the concern in our situation.
Conclusion
There was a steep curve in figuring out how to install and properly use phpdbg on our servers, but in the end, saving over 2 hours per run and allowing ourselves to gate deploys to the develop server environment based on quality and security in this way made the effort totally worth it. The biggest struggle in this process was the lack of documentation out there on phpdbg, so hopefully this article helps others who may be in the same boat!
Leave a Reply