How do you develop odoo reports? Are you testing them?

This blog post is inspired from my talksI've presented during the OCA code sprint. and at Anybox summit.
No test
Before to see how I now develop my report let's take a while to speak how was the life before!
Developing reports
Once upon a time I was getting bored while developing odoo reports for each change (Assuming starting from an existing report) I had to:
Add few modifications.
Start odoo and updating my module (Sometimes it happen your module do not update properly but sometimes you don't even notice it at that point).
Note
There is a way to auto-reload xml files with recent version of odoo, I don't really know why I'm not using it that much, probably because my IDE auto-save files which make to much breaks while auto reloading partial works.
Open a browser.
Go to the expected form view (In some case reports are stored so you have to create a new sample object each time or restore your database with a tool like odb or disengage storing the report...).
Generate the report (a lot of time the report do not generate here).
Open pdf file (at some point you get lost with several files in your download folder).
Note
I know there are some solutions to not generate the pdf and see the result directly in the browser I use it a bit at the begining but not that much, I like to see the final result with break pages, the expected width and so on.
Try to make sure to see what are changes on the PDF (which can be quite hard in some circumstances trying to open the report before and after switching windows or moving then side by side).
This was really time consuming and require a lot of manipulations switching from the console, my IDE, the browser and a pdf reader.
Testing regression
At Anybox we love quality service and apply Test Driven Development (TDD) as much as possible. So delivering a new version and hear the phone ring few hours after publishing it always makes my skin crawl. I'm pretty sure customers love to find regressions (I was one of those when I was on the other side! It's today obvious for me some customer that follow development know/detect impact of a requested evolution probably more than the developer but this is an other story).
As a developer I love to reuse the same code over my code base which can becomes hard to follow when you manage to share style sheet over your reports.
Testing report generation
To solve those issues I've started to generate reports (without taking care of the result), This resolved some of my issues:
- Before generating the report from the browser I was pretty sure it will works.
- I get less regression over the time if my model change that break a report I'm alerted by CI bot.
With some drawbacks:
- On CI I had to make sure all the environment is present (likes wkhtmltopdf / libreoffice stateless server...)
Testing reports
Next step was more audacious: compare generated report with a reference pdf file. For that we have developed the library odoo report testing which provides tools to generate and compare pdf files pixel by pixel.
So while developing your report you can focus on your development and changes without getting your mind lost in various tools switching windows.
Basically while developing a report I use 3 chained commands that:
- Update my module
- If previous command passed launch the unit test that generates the report and compare the file to a reference pdf.
- Last command open the diff file to make sure I get the expected change.
Using the anybox odoo recipe (video presentation) it looks like:bin/start_odoo -d mydb --stop-after-init -u mymodule && \ bin/nosetests -d mydb -- -s -v my_addons/mymodule/test/test_module.py:TestReport.test_report; \ gnome-open my_addons/mymodule/test/test_report_generated_compared_page_01_diff.png

As you can see diff file allows to focus on changes. Once you are happy with a change the generated pdf becomes your reference and so on.
Comparing report let you:
- save time while developing (no needs to open your browser / less switching between windows)
- detect style changes
- focus on changes and detect them quickly
- prevent any kind of regression
They are some drawbacks:
- Your report has to be predictable, you must manage in your tests: dates, sorting lines...
- you have to make sure your bot use the same environment of your production (wkhtml2pdf version...)
- we use a patched version of odoothat allows to properly load all css files even if odoo server is not responding over http
Where to go now?
If you want to see an example of how to use it read the quickstart documentation.
To understand how it works click this link!