Could you provide your performance reports and your XMT configuration (screenshot)?

It's been tested via P3 with a promising result (XMT takes less than 0.025s).
Here is the P3 report of XMT and some other plugins performance:

You can also know the plugin performance by adding ?xmt_debug_show at the end of your URL.
As you can see, XMT costs 0s (means less than 0.5s so that it's rounded to 0s)
Here is the debug result:

I noticed maybe the same problem. First used a different twitter widget, but after analysing with P3 choose for xhanch twitter. However, after a few days I noticed that if the site was inactive for a few hours it was having 85 queries and a load time of 2 to 4 seconds. Normal would be 45queries and a load time of 0,650 seconds.

Installed the plugin 'debug queries' which shows me the queries and waited a few hours and it came up with the next output.

The following lines were extra the first load (compared to normal) (btw, I changed info where it had a reference to some information about my site):

Query: update wpdb_xmt_acc set las_twt_imp_dtp = 1334754929 where nme = 'Primary'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp
Time: 0.00103306770325
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '2345234523452345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 0.000869989395142
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000878095626831
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.00031590461731
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '2345234523345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 0.000225067138672
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000492095947266
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000296115875244
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '234523452345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 8.39233398438E-5
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000528812408447
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000406980514526
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '2345234523452345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 8.41617584229E-5
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.00049901008606
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000444889068604
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '234523452345234'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 8.9168548584E-5
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000493049621582
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000295877456665
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '23452345234523'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 8.60691070557E-5
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000476121902466
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000288009643555
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '234523452345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 8.70227813721E-5
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000495910644531
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000296115875244
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '234523452345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_twt_ins
Time: 8.60691070557E-5
Query: select count(*) from wpdb_xmt_ath where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000494003295898
Query: update wpdb_xmt_ath set nme = 'tweetaccount', img_url = ' ', dte_upd = now() where uid = 'tweetaccount'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_raw_imp, xmt_ath_ins
Time: 0.000550031661987
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '234523452345234'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_ins
Time: 0.000324010848999
Query: select count(*) from wpdb_xmt_ath where uid = 'twitteruser'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.000829935073853
Query: update wpdb_xmt_ath set nme = 'another twitter1', img_url = ' ', dte_upd = now() where uid = 'twitteruser'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.000351905822754
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '2345234523452343'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_ins
Time: 0.000411033630371
Query: select count(*) from wpdb_xmt_ath where uid = 'twitteruser2'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.00180506706238
Query: update wpdb_xmt_ath set nme = 'another twitter2', img_url = ' ', dte_upd = now() where uid = 'twitteruser2'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.000456094741821
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '2345234523452345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_ins
Time: 0.00036096572876
Query: select count(*) from wpdb_xmt_ath where uid = 'twitteruser2'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.00060510635376
Query: update wpdb_xmt_ath set nme = 'another twitter2', img_url = ' ', dte_upd = now() where uid = 'twitteruser2'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.000344038009644
Query: select count(*) from wpdb_xmt_twt where acc_nme = 'Primary' and twt_id = '2345234523452345'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_twt_ins
Time: 0.000100135803223
Query: select count(*) from wpdb_xmt_ath where uid = 'twitteruser2'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.000595092773438
Query: update wpdb_xmt_ath set nme = 'another twitter2', img_url = ' ', dte_upd = now() where uid = 'twitteruser2'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_imp, include, xmt_ath_ins
Time: 0.000367879867554
Query: select twt_cch, twt_cch_dtp from wpdb_xmt_acc where nme = 'Primary'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_get, xmt_twt_cch_get
Time: 0.000866889953613
Query: select twt.twt_id, twt.twt_typ, twt.twt_dtp, twt.twt, twt.twt_src, twt.twt_ath, ath.nme as ath_nme, ath.img_url from wpdb_xmt_twt twt left join wpdb_xmt_ath ath on twt.twt_ath = ath.uid where acc_nme = 'Primary' and twt_typ not in ('rfy','rty','dmg') order by twt_id desc limit 4
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_get
Time: 0.000761032104492
Query: update wpdb_xmt_acc set twt_cch = 'a:4:{i:2345234523452345;a:8:{s:4:"type";s:3:"twt";s:9:"timestamp";s:19:"15/04/2012 14:34:48";s:5:"tweet";s:75:"Tweet messageh";s:6:"author";s:12:"tweetaccount";s:11:"author_name";s:12:"tweetaccount";s:10:"author_url";s:31:" http://a0.twimg.com/profile_images/999999999/logo.jp g" target="_blank">http://a0.twimg.com/profile_images/999999999/logo.jp g"/> ";s:6:"source";s:79:"<a href="" http://www.tweetcaster.com" ;" rel="nofollow">TweetCaster for Android</a>";}i:23452345234523;a:8:{s:4:"type";s:3:"twt";s:9:"timestamp";s:19:"14/04/2012 22:26:51";s:5:"tweet";s:59:"tweet message";s:6:"author";s:12:"tweetaccount";s:11:"author_name";s:12:"tweetaccount";s:10:"author_url";s:31:" http://a0.twimg.com/profile_images/999999999/logo.jp g" target="_blank">http://a0.twimg.com/profile_images/999999999/logo.jp g"/> ";s:6:"source";s:79:"<a href="" http://www.tweetcaster.com" ;" rel="nofollow">TweetCaster for Android</a>";}i:23452345234523;a:8:{s:4:"type";s:3:"twt";s:9:"timestamp";s:19:"14/04/2012 20:14:31";s:5:"tweet";s:67:"tweet message";s:6:"author";s:12:"tweetaccount";s:11:"author_name";s:12:"tweetaccount";s:10:"author_url";s:31:" http://a0.twimg.com/profile_images/999999999/logo.jp g" target="_blank">http://a0.twimg.com/profile_images/999999999/logo.jp g"/> ";s:6:"source";s:79:"<a href="" http://www.tweetcaster.com" ;" rel="nofollow">TweetCaster for Android</a>";}i:23452345234523452345;a:8:{s:4:"type";s:3:"twt";s:9:"timestamp";s:19:"14/04/2012 19:24:00";s:5:"tweet";s:249:"RT <a href="" http://twitter.com/twitteruser3" ;" title="Follow twitteruser3" target="_blank">@twitteruser3</a>: <a href="" http://twitter.com/twitter_user" ;" title="Follow twitteruser" target="_blank">@twitteruser</a> twitter message";s:6:"author";s:12:"tweetaccount";s:11:"author_name";s:12:"tweetaccount";s:10:"author_url";s:31:" http://a0.twimg.com/profile_images/999999999/logo.jp g" target="_blank">http://a0.twimg.com/profile_images/999999999/logo.jp g"/> ";s:6:"source";s:79:"<a href="" http://www.tweetcaster.com" ;" rel="nofollow">TweetCaster for Android</a>";}}', twt_cch_dtp = 1334754931 where nme = 'Primary'
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, dynamic_sidebar, call_user_func_array, WP_Widget->display_callback, xmt_wgt->widget, widget_xmt, xmt_twt_get, xmt_twt_cch_set
Time: 0.000145196914673
Query: SET SESSION query_cache_type = 0;
Call from: require, require_once, include, get_footer, locate_template, load_template, require_once, wp_footer, do_action, call_user_func_array, Debug_Queries->the_queries, Debug_Queries->get_queries
Total query time: 0.32077s for 84 queries.Very slow compared to normal
Total num_query time: 2.946 for 85 num_queries.Very slow compared to normal
» Different values in num_query and query? - please set the constant define('SAVEQUERIES', true);in your wp-config.php
Page generated in 2.93303s, 89.06% PHP, 10.94% MySQL

Just reading about your ?xmt_debug_show option. Will do the same test again after a few hours with that option used.

edit:
Just got the results with the ?xmt_debug_show option:
- XMT: Starting to generate output -
- XMT: Import Tweets - Start - Exec time - 0 s -
- XMT: Import Tweets - Finished - Exec time - 1 s -
- XMT: Get Tweets - Start - Exec time - 1 s -
- XMT: Get Tweets - Finished - Exec time - 1 s -
- XMT: Build Body - Start - Exec time - 1 s -
- XMT: Build header - Start - Exec time - 1 s -
- XMT: Build header - Finished - Exec time - 1 s -