Letters Got Wierd After Migration From Whmcs To Blesta

[dado023]

Ok, last night i have migrated from WHMCS(5.3.x) to Blesta(3.1.3) using built in blesta importer.

 

Now, i got some weird problems with names and surnames(lets call these "words").

Apparently particular words which contain non-usa letters screwed up.

 

š đ ž č ć are 5 Croatian letters. After import is done, these are somehow getting transformed into weird characters

 

Few examples,:

 

Bećarović(original) turned into  Bećirović(after import)

Sveučilište(original) turned into SveuÄiliÅ¡te(after import)

 

 

Why do i get these incorrect words, how do i make this right? 

I assume this has to do something with database encoding or something, but i don't know how to make this do it correctly.?

 

[Chris]

I've never seen this issue before. Lets hope someone has resolution 

[Bit Bayou]

Possibility 1

The database Character set has changed during the migration.

 

 

Possibility 2

Blesta is set with a different charset when serving the web pages.

 

 

 

Both of these possible problems can cause this outcome.. either individually or together.

[dado023]

aha i see....so what are best precaution measures to take before migration to avoid this from happening?

[Bit Bayou]

I'll have to get back to you on that. It's a Saturday night where I live, and I don't currently have time to research it further.

 

Unless someone can address this issue before me, I should have more information for you in a few days. Sorry for the delay. :/

[dado023]

no problem at all, take your time, and thank you in advance 

[Paul]

Someone from our team may have more information on this on Monday, but Blesta uses UTF-8. If the characters are not UTF-8 prior to being imported, it may explain the issue.

[Greg-Mega]

It's a character encoding issue between the WHMCS. When it comes out of WHMCS's database in latin1 it looks like this:

 

'Bećarović'

 

When it's inserted into a UTF-8 charset table, you get this:

 

'Bećarović'

 

This can be demonstrated by running the following in MySQL:

 

SELECT CONVERT(_latin1 'Bećarović' USING utf8);

Not sure of a really simple fix without running some complicated MySQL queries on your database. The importer should be able to be adjusted though, forcing a UTF-8 connection to the WHMCS database by using mysql_set_charset or the PDO equivalent which i think is 'SET NAMES 'utf8' COLLATE 'utf8_general_ci;' or the below:

SET character_set_results = 'utf8', character_set_client = 'utf8', character_set_connection = 'utf8', character_set_database = 'utf8', character_set_server = 'utf8'

 and that should be enough to resolve it during the import process.

[Michael]

Another issue with WHM** blimey they can't do anything right can they, using Latin1 and not UTF-8 what's next? 

[Blesta Addons]

the import plugin should check database encoding if not utf8 it should add iconv function to the imported data ,

[Bit Bayou]

I'm looking at whmcs_migrator.php at line 2001:

	protected function decode($str) {
		if ($str === null)
			return $str;
		return html_entity_decode($str, ENT_QUOTES, "UTF-8");
	}

While I understand with PHP version 5.4 the default character set became UTF-8, the importer is assuming UTF-8 was used in your other install when it's decoding the data...

 

This isn't the "how to fix" post, just part of what I think the problem is. The importer is assuming UTF-8 when decoding, so it's turning the letters in to the odd characters.