tsiedsma
-
Posts
88 -
Joined
-
Last visited
-
Days Won
1
Posts posted by tsiedsma
-
-
Well i let the first import run overnight, nothing else was going on. The second one ran while I'm doing other tasks. Nothing resource intensive though so I dunno.
-
Results from latest import:
importStaff ----------------- importStaff took: 0.8861 seconds ----------------- importClients ----------------- importClients took: 703.2672 seconds ----------------- importContacts ----------------- importContacts took: 5.5433 seconds ----------------- importTaxes ----------------- importTaxes took: 0.009 seconds ----------------- importCurrencies ----------------- importCurrencies took: 0.089 seconds ----------------- importInvoices ----------------- importInvoices took: 318.2752 seconds ----------------- decrypted 0 values using WHMCS' custom algorithm decrypt took: 0 seconds total time took: 1028.0708 seconds Array ( [last_name] => Array ( [empty] => Please enter a last name. ) ) -
Try the attached whmcs_migrator.php file. Place it in /plugins/import_manager/components/migrators/whmcs/.
This will only run importStaff, importClients, importContacts, importTaxes, importCurrencies, and importInvoices.
You should see speed improvements on importClients and importInvoices. I'm curious how they compare to your last run.
running now...
-
The mystery price may be a price override. Since Blesta doesn't support service price overrides yet, these are imported into the package.
There shouldn't be any price overrides for that product at all. Especially from $8.95 / month to $230.10 / month...
-
Ok, checked the tblpricing
SELECT * FROM `tblpricing` LEFT JOIN `tblhosting` ON `tblhosting`.`id`=`tblpricing`.`relid` WHERE `monthly`='230.10'
That should give you some info on where it's coming from.
Absolutely, though I'll probably have you test on just clients to get a benchmark.
No results found.
The WHMCS productid is 160 so I looked for that and got these results:
SELECT * FROM `tblpricing` WHERE `type` = 'product' AND `relid` = '160'
-
Where did 230.10 come from? Check your tblpricing table. That might shed some light on the 0.00 price as well.
Do you have the gmp extension enabled on your machine? Just curious since importing clients took 3 times as long as importing invoices, and I'd expect you'd have more invoices than clients.
FYI, I've got a theory on improving performance of the importer that I'm going to put to the test today. We'll see how it goes.
Dude, I have no idea... gmp should be enabled, yes. It's in the php.ini.
As for the pricing, I'll check the table, not sure what I'm looking for.
As for your theory, want me to try it and tell you how it goes before you release a new importer? I may have the largest database of anyone here...
-
Here is my example of the pricing issues... There are several plans where Blesta has multiple prices for the one term even though only one is truly valid.
Blesta:
WHMCS:
-
Damn, still took forever.
importStaff ----------------- importStaff took: 0.253 seconds ----------------- importClients ----------------- importClients took: 934.4394 seconds ----------------- importContacts ----------------- importContacts took: 2.1911 seconds ----------------- importTaxes ----------------- importTaxes took: 0.005 seconds ----------------- importCurrencies ----------------- importCurrencies took: 0.024 seconds ----------------- importInvoices ----------------- importInvoices took: 328.3218 seconds ----------------- importTransactions ----------------- importTransactions took: 31530.5644 seconds ----------------- importPackages ----------------- importPackages took: 7.1774 seconds ----------------- importServices ----------------- importServices took: 244.932 seconds ----------------- importSupportDepartments ----------------- importSupportDepartments took: 0.07 seconds ----------------- importSupportTickets ----------------- importSupportTickets took: 51.706 seconds ----------------- importMisc ----------------- mail log took: 3061.2902 seconds settings took: 0.659 seconds calendar events took: 0.001 seconds importMisc took: 3061.9542 seconds ----------------- decrypted 10522 values using WHMCS' custom algorithm decrypt took: 1.3451 seconds total time took: 36161.6394 seconds Array ( [last_name] => Array ( [empty] => Please enter a last name. ) ) -
Have a look at that thread please mate.
Thanks for the link, interesting requirement. Works now.
-
You need to edit your c://windows/system32/drivers/etc/hosts file and put your hostname to your IP I believe.
Why can't I just connect to http://localhost/blesta ???
I'm trying to start a new trial to just do the import with my core i7 and SSD drive so I have no performance related issues.
-
Ugh, I can't even get that to work on XAMPP on Windows.
Is there a way to find out what the problem is?
I figured I would let my core i7 do the heavy lifting, then copy the database to the server to see how things look.
-
Judging by the time it takes to run, I'd say your WHMCS database contains about 900,000 records, or your server is slower than my 3 GHz Pentium D (circa 2005). You might consider importing into a database on a separate server than you are importing from. That is, if WHMCS database is on server1, have Blesta database on server2. This will help split the CPU/disk load. Though this really only makes sense if both server1 and server2 are on the same LAN, or are connected through a 100mbps+ connection. You might also consider performing the import on faster hardware (CPU/disk).
Blesta does not create temporary tables at all, so sounds like you might also have something else running on the same machine slowing things down.
I'm gonna try it on my computer instead of my dev VPS. Any chance I can get a Dev license for Blesta without having an active license?
-
temp tables may be created automatically by mysql
You're right, I did some digging and there is nothing in the blesta code that does any of the advanced stuff I saw in the SQL logs. Either way, the queries were definitely related to processing transactions which took a majority of the time.
Since my WHMCS license is owned, I may keep it around as an archive. I can always modify the client area pages in Blesta to pull in data via the WHMCS API such as invoice history and stuff so I don't have to worry so much about importing historic data.
-
I am using the latest version of the importer, after I received that error I checked my whmcs database and couldn't find any missing first or last names.
Not sure where the error came from but it happened twice.
The other issue is the length of time the importer takes. Perhaps it's because of how WHMCS does things, maybe because of how you do things.
I watched the queries running on the server and it appears that when you apply transactions there are temporary tables created and this takes a boatload of time.
I have yet to see a complete import even with a seriously reduced WHMCS database.
-
I had a migration running for most of the day, it completed with an error "Please enter a last name".
execution time: 26777.006063 seconds (7.43806 hours) LOL
That was a full database import, it did get the packages but not the client packages.
It also didn't get the support tickets this time.
I just ran another import, well it just finished anyways. Even with your default values and the fact that I set all empty last names to Smith, I still got the error:
Almost everything imported for my test group.
A couple of issues.
- Servers are listed but show 0 accounts.
- Editing and saving the server details causes the number of accounts to populate
- Server default name servers were not imported
- Server Label and Server Hostname are the same, WHMCS has Server Name and Server Hostname which I would expect to be imported respectively.
- Support stuff came over but there is no Support option in the header even though the support manager is installed.
- (Default Admin group doesn't get assigned permissions to the support manager)
- Some packages have odd pricing. I'm seeing some with duplicate terms like Monthly but with two prices and only one is accurate.
- Some packages have quantity available set to zero even though WHMCS had stock control unchecked.
- All tickets came over as Open and the Ticket Number doesn't show up in the Ticket list.
- Servers are listed but show 0 accounts.
-
Is there any update on the outstanding issues?
-
Looks like ours is 20% faster.
I was wrong about the original benchmarks, however. Their algorithm is not terribly inefficient (despite it being terribly insecure). I've updated the debugging in the next rev to accurately reflect the proper calculations.
LOL, I love this. Your function that you hacked together to decrypt their data is faster than the one they use. That just proves how much better at this you are than them.
-
So... I've got some initial stats on the migrator.
On my machine each DB write takes .03 seconds on average, so 1000 records = 30 seconds. Hopefully your machine is faster, but likely your dataset is much larger. Just something to keep in mind.
The biggest slowdown is WHMCS' custom cipher algorithm. In case you weren't aware, WHMCS encrypts credit card numbers using AES (good for them), but they encrypt everything else using their stupid custom encryption algorithm which is nothing more than a glorified caesar cipher. Well, we cracked the algorithm a while ago and implemented it as part of the importer and it turns out it's slow as molasses.
That's 0.383 seconds per string! In contrast, our AES encryption takes 0.0007 seconds per string.
WHMCS' insecure custom algorithm takes 550 times longer to run than 256-bit AES.
EDIT: Factoid, the full import took 39.2339 seconds. Over 25% of the time spent was attempting to decrypt WHMCS encrypted data.
Have you benchmarked WHMCS decryption using the WHMCS decrypt function? Just curious if you found that their function was faster or slower than what you came up with.
If you take the whmcs/includes/functions.php file, you can run the decrypt function. You'll need the $cc_encryption_hash variable defined before running the function as they appear to globalize that variable and use it in the function. I know this because it doesn't work without it in the script.
<?php include('whmcs/includes/functions.php'); $cc_encryption_hash = '1234'; $password = decrypt($encryptedpassword); ?>Edit:
You probably already know this and have even seen the source since a quick search on your favorite pastebin type site for anything WHMCS related returns lots of results for decoded files and functions...
-
For those having issues with ticket status values matching up, what are the results of the following query on your WHMCS db?
SELECT status, COUNT(*) AS total FROM tbltickets GROUP BY status
Note, the 'total' values don't really matter, but would be good to see in contrast to what's you're seeing in Blesta.
On my smaller test database:
Answered 5 Closed 4731 In Progress 1
My actual database has much much more. The test import showed all of them as Open.
-
Tickets should come across with an equivalent status as in WHMCS. Here's the mapping, on the left are status values in WHMCS, on the right are their status values in Blesta. If a ticket has a status not listed on the left Blesta marks it as "open".
'Open' => 'open', 'Answered' => 'closed', 'Customer-Reply' => 'awaiting_reply', 'Closed' => 'closed', 'In Progress' => 'in_progress'
In my testing, that isn't working.
-
I had a migration running for most of the day, it completed with an error "Please enter a last name".
execution time: 26777.006063 seconds (7.43806 hours) LOL
That was a full database import, it did get the packages but not the client packages.
It also didn't get the support tickets this time.
-
I've noticed one thing around here and that is that everyone seems to have the a module, plugin, importer, feature that they need. Sometimes there is this insinuation that the devs aren't doing enough or list of things which they 'should' be doing. They can't just drop everything on focus on just one client with their specific needs alone (show me one company that does?). Mind you that it's a client management and billing system. It has an open modules/plugin system which means you can develop or have someone develop your modules. You can also have your own importer build. But if you're going to wait around for the complimentary modules and importers that come at no cost to you then a little patience are in order. Especially if your going to critique them about their so-called 'behind-the-scenes problems' as someone described it. If it was so easy and you want get migrated so badly why wouldn't you just build it yourself? Nothing is stopping you.
Blesta is build with quality in mind and this takes time. The community couldn't be more helpful.
I don't mean help this topic run away but good grief! It had to be said.
True, there will always be those people with the attitude of "If you want my business, you will have to bend over backwards to earn it. But if you screw up, I'm out the door and onto the next". Anyone that's been in business long enough will tell you that those are not the customers you want which is why those people in this thread are not getting the attention they think they are entitled to.
-
Agreed, and it might be a good idea for one of the Blesta crew to update the OP to say that this is a work in progress. Otherwise someone might download the latest beta, have it not work, conclude that the system is broken, and not realize that work is ongoing.
I guess not everyone understands what Beta means.
be·ta noun
a version of a product (such as a computer program) that is almost finished and that is used for testing
-
As it stands, I cannot import my actual production database until this importer can be optimized to run in a shorter time window, or until it can be done in segments.
It doesn't even get through all of the transaction processing before it dies. I have Apache, MySQL and PHP all set with timeout values of 18,000 seconds. That's 5 hours and should be more than enough time.
This is a test server, so those values are ok for me but not everyone will be able to or want to make those changes on their servers.
I can't be the only one with this large of a database, I have been using WHMCS for 5 or 6 years so thats alot of invoices, transactions, emails and clients.
Whmcs Migrator (Beta) - Updated 2013-11-12
in The Lounge
Posted
I agree, an option would be great. It would also be nice to support running the importer to import staff and clients, then run it later to import packages / products / services. Then run it again and just import misc, or support tickets or whatever.
Obviously some items require others to exist, you can't import invoices without clients and so on. I'd also like an option to not import clients that don't have active services or domains. Since you don't have an affiliate system, there is no reason for me to import users in my system that are no longer active or don't have any services.