Support Department Imap Not Working

[drolfe]

HI, 

 

I've setup a support department, using google apps for email, set to IMAP with the attached settings.

 

I have an email reply from a ticket (from a client email) but it's not pulled into blesta ?

 

How can I troubleshoot this ?

 

Regards, Daniel

 

 

[drolfe]

Ok I think I have this working,

 

After looking at the database and how the cron works:

 

note that run ID 19 is "download tickets" as below:

mysql>  SELECT * FROM cron_tasks WHERE `id` = 19;
+----+--------------+-----------------+------------------+------------------------------------------------------------------------------------+---------+----------+
| id | key          | plugin_dir      | name             | description                                                                        | is_lang | type     |
+----+--------------+-----------------+------------------+------------------------------------------------------------------------------------+---------+----------+
| 19 | poll_tickets | support_manager | Download Tickets | Connects to the POP3/IMAP server to download emails and convert them into tickets. |       0 | interval |
+----+--------------+-----------------+------------------+------------------------------------------------------------------------------------+---------+----------+
1 row in set (0.00 sec)
mysql> 

Then I found in another thread to look for cron_logs without a finish time: 

mysql> SELECT * FROM `log_cron` WHERE `run_id` !=0 AND `end_date` IS NULL ORDER BY `start_date` DESC;
+--------+-------+----------------------------------+----------------------------------------------------------+---------------------+----------+
| run_id | event | group                            | output                                                   | start_date          | end_date |
+--------+-------+----------------------------------+----------------------------------------------------------+---------------------+----------+
|     19 |       | bfcafb10b07f574e429303fd1b4f46a8 | Attempting plugin cron for support_manager poll_tickets. | 2015-02-23 09:20:02 | NULL     |
|     19 |       | 5706097d22923690b0d0eed4effd07dd | Attempting plugin cron for support_manager poll_tickets. | 2015-02-23 03:20:01 | NULL     |
|     19 |       | d9ac43b45eee404ef29fe0a03478f484 | Attempting plugin cron for support_manager poll_tickets. | 2015-02-22 20:55:01 | NULL     |
|     19 |       | 2aa266c0262becd70a1ea157fd6c8368 | Attempting plugin cron for support_manager poll_tickets. | 2015-02-22 10:35:02 | NULL     |
|     19 |       | a303667c33f8715ec85957819c00f42a | Attempting plugin cron for support_manager poll_tickets. | 2015-02-22 04:30:02 | NULL     |
+--------+-------+----------------------------------+----------------------------------------------------------+---------------------+----------+
5 rows in set (0.01 sec)
mysql>

Then deleting them:

mysql> DELETE FROM `log_cron` WHERE `run_id` !=0 AND `end_date` IS NULL ORDER BY `start_date` DESC;
Query OK, 5 rows affected (0.04 sec)

mysql> SELECT * FROM `log_cron` WHERE `run_id` !=0 AND `end_date` IS NULL ORDER BY `start_date` DESC;
Empty set (0.01 sec)

Now when the cron runs I can see the connection to google imap

root@blesta:~# tcpdump -npi eth0 port 993
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

15:35:01.840128 IP 192.168.10.114.42386 > 74.125.136.109.993: Flags [S], seq 2150746609, win 29200, options [mss 1460,sackOK,TS val 2057746 ecr 0,nop,wscale 7], length 0
15:35:01.846315 IP 74.125.136.109.993 > 192.168.10.114.42386: Flags [S.], seq 2080383863, ack 2150746610, win 42540, options [mss 1380,sackOK,TS val 1611920504 ecr 2057746,nop,wscale 7], length 0
15:35:01.846361 IP 192.168.10.114.42386 > 74.125.136.109.993: Flags [.], ack 1, win 229, options [nop,nop,TS val 2057747 ecr 1611920504], length 0
15:35:01.846801 IP 192.168.10.114.42386 > 74.125.136.109.993: Flags [P.], seq 1:306, ack 1, win 229, options [nop,nop,TS val 2057748 ecr 1611920504], length 305
15:35:01.852916 IP 74.125.136.109.993 > 192.168.10.114.42386: Flags [.], ack 306, win 341, options [nop,nop,TS val 1611920511 ecr 2057748], length 0
15:35:01.854500 IP 74.125.136.109.993 > 192.168.10.114.42386: Flags [.], seq 1:2737, ack 306, win 341, options [nop,nop,TS val 1611920512 ecr 2057748], length 2736
15:35:01.854549 IP 192.168.10.114.42386 > 74.125.136.109.993: Flags [.], ack 2737, win 271, options [nop,nop,TS val 2057750 ecr 1611920512], length 0

........ output cut

77 packets captured
77 packets received by filter
0 packets dropped by kernel
root@blesta:~# 

I can also see the email import is now working :-)

 

Please see attached / below the final working settings for google apps hosted domain IMAP, don't forget to enable IMAP support under your google account also.

 

Regards, Daniel

[Michael]

Any errors? I don't like using IMAP myself and use Piping.

[drolfe]

HI,

 

I can't see any errors, but again I don't really know where to look.

 

I'm happy to use any method, I just can't seem to figure it out, I'm not using any panel, just google apps hosted email domian

 

Any help would be great

 

Regards, Daniel

[Michael]

Incoming Mail (IMAP) Server - Requires SSL
imap.gmail.com
Port: 993
Requires SSL: Yes

Outgoing Mail (SMTP) Server - Requires TLS
smtp.gmail.com
Port: 465 or 587
Requires SSL: Yes
Requires authentication: Yes
Use same settings as incoming mail server

 

So I would say the above has to be in the ticket bit which you have done, but have you done the outgoing email? Settings > Company > Emails > Mail Settings

 

Does SMTP work for you? You can select that from the dropdown and use the above information.

 

Has the ports been opened in the firewall on the server? (If you aren't hosting yourself and using a shared / reseller) ask your provider.

 

Any logs in Tools > Logs > Email?

 

As I said I only prefer Piping which means you don't use another provider to send emails just your server's mail server itself.

[serge]

IMAP working fine for me but I'm using my own mail-server.

 

Check :

- your firewall to see if port you need are open

- read google documentation about protocol to use & port

[drolfe]

Hi, Yes outgoing is working, see attached settings.

 

No firewall, blesta is hosted on our own servers

 

Regards, Daniel

[Michael]

Erm have you assigned yourself to the departments? I know that sounds silly but I managed to forget once haha. 

 

If not I would try another provider and see if that works but the guys are off until Monday.

[drolfe]

Is there an email import log or similar ?

 

I'm basically using the exact same settings as our whmcs install which we are trying to move away from

 

Regards, Daniel

[drolfe]

Hmmm interesting, the email 5 minute check job under automation seems to be stuck, note the other 5 minute checks are running normally

 

see attached

[Michael]

Hmmm interesting, the email 5 minute check job under automation seems to be stuck, note the other 5 minute checks are running normally

 

see attached

 

Nice find, ok let it run for a while then visit it and click unlock.

[serge]

No firewall, blesta is hosted on our own servers

Really a server without firewall?

 

Maybe you should get a firewall, abuser ip should be banned, etc, it's very unsafe a server without firewall.

You have maybe one active without being aware, if not this free "CSF firewall" is really good

[drolfe]

No firewall, blesta is hosted on our own servers

Really a server without firewall?

 

Maybe you should get a firewall, abuser ip should be banned, etc, it's very unsafe a server without firewall.

You have maybe one active without being aware, if not this free "CSF firewall" is really good

 

 

HI,

 

Yes there is a firewall, not on the server, the VM is behind an ASA firewall, IP any any outbound is allowed, only inbound ports are restricted.

 

Regards, Daniel

[Paul]

The email and the user for the support department are different. Email can only be imported to the department if the TO address matches that of the department email. Since you're authenticating with Google using a different address, I wonder if this might be part of the issue.

 

If the ticket import is stalling there may be other issues. Please ensure that the mailparse pecl extension is installed. Also, you could disable your cron job, wait 5 minutes, email the ticket address, and then run the cron manually under Settings > System > Automation. This won't work if the task is still hung (spinning icon under Settings > Company > Automation). This should show you the ticket cron output. If it seems to stall, or still doesn't work and the task shows that it is locked, try enabling error reporting and running the cron again (after at least 5 minutes).

 

To enable error reporting, edit /config/blesta.php and change

 

Configure::errorReporting(0);

 

to

 

Configure::errorReporting(-1);

[drolfe]

HI, Sorry ,

 

I just can't get this task to stop loading in the webui no matter what I do, Ive disabled the job, disabled cron , restarted it just keeps running, I've also used the standard email address.

 

 

As far as I know mailparse is installed:

root@blesta:~# grep -i mailparse /etc/php5/cli/php.ini 
extension=mailparse.so
root@blesta:~# grep -i mailparse /etc/php5/fpm/php.ini 
extension=mailparse.so
root@blesta:~# locate mailparse.so
/usr/lib/php5/20121212/mailparse.so
root@blesta:~# 

Error reporting is also now enabled but I don't see any more details anywhere, where should I be looking ?

 

 

See current cron output:

PHP Notice:  Constant HTACCESS already defined in /var/www/XXXXXX/htdocs/lib/init.php on line 44
PHP Stack trace:
PHP   1. {main}() /var/www/XXXXX/htdocs/index.php:0
PHP   2. include() /var/www/XXXXX/htdocs/index.php:17
PHP   3. define() /var/www/XXXXXX/htdocs/lib/init.php:44
Attempting to run all tasks for XXXXX
Attempting to apply credits to open invoices.
There are no invoices to which credits may be applied.
The apply credits task has completed.
Attempting to deliver invoices scheduled for delivery.
No invoices are scheduled to be delivered.
The deliver invoices task has completed.
Attempting to provision paid pending services.
The paid pending services task has completed.
Attempting to unsuspend paid suspended services.
The unsuspend services task has completed.
Attempting to process renewing services.
The process renewing services task has completed.
Attempting plugin cron for order accept_paid_orders.
Finished plugin cron for order accept_paid_orders.
All tasks have been completed.
Attempting to run all system tasks.
All system tasks have been completed.

See the job issue attached:

 

Regards, Daniel

[serge]

I suspect your :

 

- firewall parameters are wrong: incoming port not open

or

- wrong imap account credential

 

So in first case you can get timeout with email server, but I have noticed, even in such case Blesta can strangely say in cron output

 

"Attempting to run all system tasks.
All system tasks have been completed."

 

 

And you need to correct issue, & purge the record in the cron log table at Blesta database

 

You was saying you have no firewall restriction for outgoing communication, but please think also at in-coming & allowed port because you need it for email account authentication....

[drolfe]

& purge the record in the cron log table at Blesta database

 

 

directly via mysql client ?

 

 

 

no much point checking the firewall just yet as the blesta node isn't even trying to connect out

root@blesta:~# tcpdump -npi eth0 port 995
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@blesta:~# 

note I'm currently trying pop via gmail, it's not working either

 

Regards, Daniel

[serge]

Sql query to remove all log cron record is

 

But select your blesta database before

 

---------------

 

truncate log_cron;

 

-------------

 

and after run once the cron manually from blesta interface at Settings/System/automation

 

 

Doing that way, make any issue with cron task should be "removed/unlocked" , and cron will run normally after with the scheduled cron

[Michael]

Glad you managed to get it working I wouldn't have ever guessed that.

[serge]

Great,

 

So in my opinion, there is also a bug in Blesta because, a cron task was never ending (email server timeout, etc) but there was such confirmation

"All system tasks have been completed." when it's was real false.

 

I found it before on my Blesta and here it was the same

[Tyson]

So in my opinion, there is also a bug in Blesta because, a cron task was never ending (email server timeout, etc) but there was such confirmation

"All system tasks have been completed." when it's was real false.

 

Each cron task that runs also logs that it has started and that it has ended, regardless of whether the task outputs anything else about what it did.

 

The system message: "All system tasks have been completed" simply indicates that all of the system tasks that were scheduled to be run have been run. If any of those system tasks were unsuccessful, or produced an error, you should check the log for those particular cron tasks to see why.

[serge]

yes, I know what you said is reflecting how Blesta is now working.

 

BUT using very common sense how can we really display, "all cron tasks completes" when finally one of them did not ended?

 

It's in my opinion very antagonist of the common meaning,

[Paul]

yes, I know what you said is reflecting how Blesta is now working.

 

BUT using very common sense how can we really display, "all cron tasks completes" when finally one of them did not ended?

 

It's in my opinion very antagonist of the common meaning,

 

I would suggest a feature request for better reporting on failed tasks. Personally, I want to build a notification system in Blesta that would allow notices to be sent to staff, and they would pop up/slide up in the corner of the screen. Something like this could potentially be useful in notifying high level staff that a particular task failed. Also useful for failed module requests, new orders, new tickets, and more.

[Michael]

I would suggest a feature request for better reporting on failed tasks. Personally, I want to build a notification system in Blesta that would allow notices to be sent to staff, and they would pop up/slide up in the corner of the screen. Something like this could potentially be useful in notifying high level staff that a particular task failed. Also useful for failed module requests, new orders, new tickets, and more.

 

Paul one word... Yum! haha sounds so wicked!

[Tyson]

yes, I know what you said is reflecting how Blesta is now working.

 

BUT using very common sense how can we really display, "all cron tasks completes" when finally one of them did not ended?

 

It's in my opinion very antagonist of the common meaning,

 

The specific task that failed would not say that it had completed, which is how you would know that it did not end. Each task begins and ends with a log message (e.g. "Attempting to..." and "...task has completed").

 

The cron is setup for tasks to be grouped into their respective company or system tasks. The group also begins and ends with a log message (e.g. "Attempting to run all system tasks" and "All system tasks have been completed.") because it can be useful to know if the group has run, whether or not any of the tasks in that group had run too.

 

e.g.   

Attempting to run all tasks for XXXXX
   Attempting to apply credits to open invoices.
      There are no invoices to which credits may be applied.
   The apply credits task has completed.
   
   Attempting to deliver invoices scheduled for delivery.
      No invoices are scheduled to be delivered.
   The deliver invoices task has completed.
   
   Attempting to provision paid pending services.
   The paid pending services task has completed.
   
   Attempting to unsuspend paid suspended services.
   The unsuspend services task has completed.
   
   Attempting to process renewing services.
   The process renewing services task has completed.
   
   Attempting plugin cron for order accept_paid_orders.
   Finished plugin cron for order accept_paid_orders.
All tasks have been completed.

Attempting to run all system tasks.
All system tasks have been completed.