Blesta generated thousands of duplicate invoices, but I think I figured out why

[Panormitis]

Hello,

I'm using Blesta 5.11.3 with PHP 8.3 and OpenLiteSpeed.

I'm using also Domain Manager and the Enom domain module. I have activated most TLDs and set TLD pricing sync as well.

Out of the blue, one of the customers contacts me and says his Blesta account is filled with duplicate unpaid invoices.

I checked the customer's account and he had about 3000 unpaid invoices and Blesta was keep generating more... The same thing was happening to a second customer too. Both of them had one past due invoice, I don't know if that was relevant.

When I was stopping the cronjob, the generation of duplicate invoices stopped too, when I was resuming the cronjob the generation of invoices resumed too.

I checked the cron logs and it seems the cronjob was taking too long to complete because of TLD pricing sync for the TLDs and essentially the cronjob was never fully completed.

The command ps -aux | grep cron was also showing multiple cronjobs of Blesta running simultaneously.

 

At this point everything was a huge mess so I decided to restore a backup.

I killed all the cronjob processes and then I went to Settings -> Company -> Automation -> Plugin and unchecked "Domain TLD Synchronization - Synchronize packages with the TLD pricing from their registrar module"

Finally the cronjob was able to fully run:

Accept Paid Pending Orders
Download Tickets
Close Tickets
Build Mailing Export
Send Mass Email
Delete Trash Tickets
Send Reminders
Process Abandoned Orders
Transition Quotations
Apply Payments to Open Invoices
Provision Paid Pending Services
Deliver Invoices
Exchange Rate Updates
Process Service Renewals
Unsuspend Services
Process Service Changes

However I also wanted TLD pricing synchronization to work, without messing anything up, so I decided to make a script to run it as a separate cronjob:

<?php
// Blesta environment location.
define("BLESTA_DIR", "/home/user/domains/domain.com/public_html/blesta");
require BLESTA_DIR . "/vendors/autoload.php";
require BLESTA_DIR . "/index.php";

// Load the TldSync class
require_once BLESTA_DIR . "/plugins/domains/lib/tld_sync.php";

// Initialize the TldSync tool
$tldSync = new TldSync();

// Optional: define which TLDs to sync. Empty = all.
$tlds = [];  // e.g., ['.com', '.net']

// Optional: filter by module (e.g., only Enom), or by terms
$filters = [
    // 'module_id' => 3,
    // 'terms' => [1, 2]
];

// Try to sync the prices...
try {
    $tldSync->synchronizePrices($tlds, Configure::get("Blesta.company_id"), $filters);
    echo "Pricing synchronization of TLDs completed successfully!";
} catch (Throwable $e) {
    echo "Error during pricing synchronization: " . $e->getMessage() . "\n";
}

I saved it as sync_tld_prices.php one level above public_html because I don't want it to be accessible by visitors and added a cronjob every 48 hours for it:

	0	1	*/2	*	*	/usr/local/bin/php -q /home/user/domains/domain.com/sync_tld_prices.php

So far I haven't seen any duplicate invoices again, so hopefully the situation is resolved.

[Paul]

When a service or domain is invoiced for renewal, the services.renew_date is bumped forward (+ term/period, e.g. 1 year, or 1 month, etc), which prevents it from being invoiced again because Blesta is only looking for services set to renew within a set period of time. I'm curious whether the renew_date was being bumped forward on this service/domain or not. Did all the invoices show the same period/date range on the line items? My guess is that something was preventing the renew date from being bumped forward, there may be some errors in your ../logs_blesta/ logs that could give a clue.

Overlapping cron jobs are normally not a problem because Blesta locks the task when it starts it by logging the attempt, then when it's finished it logs the end datetime. Until the end datetime is set, it should not run again. However, it is possible that if multiple cron jobs start at exactly the same time, this may not be detected. In that case you will find in the log_cron table multiple runs that start at the same time and overlap. You should confirm that you have only a single cron job set up for Blesta.

The TLD sync task should not interfere with other tasks and we have no other reports of this. This is because each individual task is locked while it is running and cannot run again until the previous run finishes. If the TLD sync task is taking a long time, then you could have another Blesta cron task run, but it should only process tasks that are not currently running and should be started again.

Can you provide more information?

[Panormitis]

Most likely the TLD sync task was halting all the other tasks. Before this happened, the customer purchased a service for 23€. What I assume, is that the cronjob never fully completed all the tasks regarding the purchase/invoice and while the other cronjobs were running in parallel, kept generating invoices for the same thing (while being halted again by the TLD sync task), since they thought the procedure was never finalized.

Since I have excluded the TLD Sync task and run it via a separate cronjob (which takes 2 hours to complete because it's synchronizing 462 entries), I had no further issues so far.

This generated around 11000 duplicate invoices in total, so it was a huge mess to even keep logs or files, so I decided to remove everything (logs/files/database) and restore a weekly backup and then find a solution.

[Panormitis]

Paul the issue is happening again, I killed the cronjobs and stopped the cronjob from running further.

I have the gathered the logs this time. I'll open a ticket in blesta.com for further investigation.

The only thing I did is I re enabled maldetect's cron 10 minutes ago, but I don't think that's relevant

[Panormitis]

Paul (or anyone else from Blesta stuff) please check ticket #9573368

[Paul]

Your logs show no errors, but the info log does show the invoice generation. I've replied to your ticket for more information, but this is consistent with what I said earlier that the renew_date may not be being bumped forward which should happen automatically after invoice creation. Any 3rd party extensions or customizations will be suspect, particularly any that tie into events.

[Paul]

Also, check Tools > Logs: Cron tab filtering by the word "Create" to list the Create Invoice tasks. I want to know the frequency of this task running (Start Date / End Date) as it should be a full day apart, and click the records to expand to show the output from the run. I want to know if the task is running more often than once a day and if the invoices are created in duplicate across separate runs or within the same run.